gga_load_data issueshttps://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues2023-05-11T15:28:41+02:00https://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/17Elasticsearch cross research2023-05-11T15:28:41+02:00Arthur Le BarsElasticsearch cross research- Elasticsearch
- https://www.elastic.co/guide/en/elasticsearch/reference/current/add-elasticsearch-nodes.html
- https://logz.io/blog/elasticsearch-cluster-tutorial/
- https://elasticsearch-py.readthedocs.io/en/v8.4.3/
View index:
- h...- Elasticsearch
- https://www.elastic.co/guide/en/elasticsearch/reference/current/add-elasticsearch-nodes.html
- https://logz.io/blog/elasticsearch-cluster-tutorial/
- https://elasticsearch-py.readthedocs.io/en/v8.4.3/
View index:
- https://stackoverflow.com/questions/14565888/how-can-i-view-the-contents-of-an-elasticsearch-index
Create cluster:
- https://logz.io/blog/elasticsearch-cluster-tutorial/
Issue about support on github:
- https://github.com/elastic/elasticsearch-docker/issues/91Arthur Le BarsArthur Le Barshttps://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/16Ensure datasets are correctly uploaded to history without duplicates2022-09-01T15:54:42+02:00Arthur Le BarsEnsure datasets are correctly uploaded to history without duplicates- [ ] Implement in `add_dataset_from_library` (gga_load_data)- [ ] Implement in `add_dataset_from_library` (gga_load_data)Arthur Le BarsArthur Le Barshttps://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/15Handle 2 organism per species for rnaseq tracks workflow2022-09-01T13:40:09+02:00Arthur Le BarsHandle 2 organism per species for rnaseq tracks workflow- [x] Verify that running the workflow for each organism works to update jbrowse "sans se marcher sur les pieds". If not, create 2 workflows: 1 for 1 organism, 1 for 2 organisms (like other workflow invocations)- [x] Verify that running the workflow for each organism works to update jbrowse "sans se marcher sur les pieds". If not, create 2 workflows: 1 for 1 organism, 1 for 2 organisms (like other workflow invocations)Arthur Le BarsArthur Le Barshttps://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/14Add step "Add organisms to jbrowse container" in rnaseq tracks workflow2022-09-01T13:39:17+02:00Arthur Le BarsAdd step "Add organisms to jbrowse container" in rnaseq tracks workflowThe following discussion from !45 should be addressed:
- [x] @lgueguen started a [discussion](https://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/merge_requests/45#note_6468):
> Missing step `Add organisms to JBrowse contai...The following discussion from !45 should be addressed:
- [x] @lgueguen started a [discussion](https://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/merge_requests/45#note_6468):
> Missing step `Add organisms to JBrowse container` to load the new jbrowse into the container
- [x] Add unique_id and display_name to render_vars in job file templating
- [x] Update workflow in constants.pyArthur Le BarsArthur Le Barshttps://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/13Store planemo job files in a "planemo" dir in organisms directory trees (inst...2022-09-01T11:11:04+02:00Arthur Le BarsStore planemo job files in a "planemo" dir in organisms directory trees (instead of current script dir)Arthur Le BarsArthur Le Barshttps://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/12Filter jbrowse dataset using constant JBROWSE_TOOL_ID (get_datasets)2022-09-01T13:47:47+02:00Arthur Le BarsFilter jbrowse dataset using constant JBROWSE_TOOL_ID (get_datasets)The following discussion from !45 should be addressed:
- [ ] @lgueguen started a [discussion](https://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/merge_requests/45#note_6433):
> Favor the filters inside the bioblend get_dat...The following discussion from !45 should be addressed:
- [ ] @lgueguen started a [discussion](https://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/merge_requests/45#note_6433):
> Favor the filters inside the bioblend get_datasets method. Use `tool_id` instead of `name` an `extension` (more reliable):
>
> `jbrowse_datasets = self.instance.datasets.get_datasets(tool_id=constants_phaeo.JBROWSE_TOOL_ID)`Arthur Le BarsArthur Le Bars2022-09-02https://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/11Add tracks directory to organisms input file and docker compose template2022-09-01T11:09:36+02:00Arthur Le BarsAdd tracks directory to organisms input file and docker compose templateThe following discussion from !45 should be addressed:
- [ ] @lgueguen started a [discussion](https://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/merge_requests/45#note_6407): (+1 comment)
> Use a new variable, and add it i...The following discussion from !45 should be addressed:
- [ ] @lgueguen started a [discussion](https://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/merge_requests/45#note_6407): (+1 comment)
> Use a new variable, and add it in the `config.yml` file
>
> `- {{ supplementary_data_directory }}:{{supplementary_data_directory }}:ro`Arthur Le BarsArthur Le Barshttps://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/9BAM and BW files as symlinks in project_data library2022-09-01T13:37:53+02:00Arthur Le BarsBAM and BW files as symlinks in project_data library### TODO
- [x] Create symlinks for BAM and BW files
- [x] Add symlinks to project data library in galaxy
- [x] Set and get datasets IDs
- [x] Template job file using datasets IDs
- [x] Invoke workflow### TODO
- [x] Create symlinks for BAM and BW files
- [x] Add symlinks to project data library in galaxy
- [x] Set and get datasets IDs
- [x] Template job file using datasets IDs
- [x] Invoke workflowArthur Le BarsArthur Le Barshttps://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/8Orthology data2022-10-13T17:47:07+02:00Loraine GueguenOrthology data### Phylotree
http://gitlab.sb-roscoff.fr/abims/e-infra/gga/tree/master_gitlab_inria_genodock_demo/swarm#orthology-data
/projet/sbr/phaeoexplorer_raw/finalresult/functional_annotation_orthofinder/Gene_Trees
- [x] Script de chargement ...### Phylotree
http://gitlab.sb-roscoff.fr/abims/e-infra/gga/tree/master_gitlab_inria_genodock_demo/swarm#orthology-data
/projet/sbr/phaeoexplorer_raw/finalresult/functional_annotation_orthofinder/Gene_Trees
- [x] Script de chargement (planemo run)
- [x] Test avec un arbre newick
- [ ] Test sur un sous-ensemble (e.g. choisir ~100-200 fichiers newick les plus petits en taille de fichier, et importer les organismes inclus dans ces fichiers)
- [ ] test sur l'archive totale (~20 000 arbres)
### GCV
- [-] Essayer de supprimer/transformer la requete necessitant un gene
- [-] Modifier la requete pour select sur le uniquename au lieu du name
- Wiki:
- https://github.com/legumeinfo/gcv/wiki
- https://github.com/legumeinfo/gcv/wiki/Services-API-v1Arthur Le BarsArthur Le Barshttps://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/7Add RNASeq tracks to JBrowse2022-09-01T11:09:27+02:00Arthur Le BarsAdd RNASeq tracks to JBrowseA partir d'une liste de fichiers bam et bw dans input.yml.
- [x] ajout automatisé des fichiers bam et bw dans input.yml (phaeoexplorer_tools)
- [x] récupération du dernier dataset "jbrowse"
- [x] planemo run avec l'outil "jbrowse" sur u...A partir d'une liste de fichiers bam et bw dans input.yml.
- [x] ajout automatisé des fichiers bam et bw dans input.yml (phaeoexplorer_tools)
- [x] récupération du dernier dataset "jbrowse"
- [x] planemo run avec l'outil "jbrowse" sur une collection de fichiers, en mode updateArthur Le BarsArthur Le Barshttps://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/6Implement planemo run2022-09-01T11:09:17+02:00Arthur Le BarsImplement planemo run## TODO
- [x] .ga workflow
- [x] planemo .yml config file (jinja2)
- [x] input parameters for each organism to configure planemo conf files## TODO
- [x] .ga workflow
- [x] planemo .yml config file (jinja2)
- [x] input parameters for each organism to configure planemo conf filesArthur Le BarsArthur Le Bars2022-02-10https://gitlab.sb-roscoff.fr/abims/e-infra/gga_load_data/-/issues/2Use short name for url and stack name2022-11-15T15:25:56+01:00Loraine GueguenUse short name for url and stack namestack ectocarpus_species4 should be ectoccarpus_sp4
https://abims-ggadev.sb-roscoff.fr/sp/ectocarpus_species4/ should be https://abims-ggadev.sb-roscoff.fr/sp/ectocarpus_sp4/
also dir name : ectocarpus_species4
Comes from input "speci...stack ectocarpus_species4 should be ectoccarpus_sp4
https://abims-ggadev.sb-roscoff.fr/sp/ectocarpus_species4/ should be https://abims-ggadev.sb-roscoff.fr/sp/ectocarpus_sp4/
also dir name : ectocarpus_species4
Comes from input "species" of "sp_dict", so from the input.yml file