Ticket #10446 (closed task: fixed)

Opened 4 months ago

Last modified 3 months ago

Y1A1 SN DFULL faltando no hadoop em testing

Reported by: julia@… Owned by: carlosadean
Priority: normal Milestone: no milestone
Component: Operation/DataManagement Severity: normal
Keywords: Cc:
Estimated Number of Hours: 0.0 Add Hours to Ticket: 0
Deviation?: yes Total Hours: 0

Description

Bom dia,

estou tendo problemas para usar o Y1A1 SN DFULL em testing. Parece que os arquivos não estão instalados no hadoop. Este dataset é de interesse para testar o workflow de GE.

Segue exemplo de falha:

 http://testing.linea.gov.br/VP/getViewProcessCon?process_id=10030566#

Se olhar no Pipeline Out do primeiro componente, verá que apesar de aparecer como sucesso, ele printa a mensagem:

Pipeline Out
( Hadoop FS )


tag_path: /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_SN_DFULL/cats


cmd: ['/hadoop-install/hadoop-2.6.0/bin/hadoop', 'fs', '-du', '-h', '/portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_SN_DFULL/cats/*.fits']


Tiles not retrieved from hadoop. Is hadoop running??

Obrigada,

Julia

Change History

comment:1 Changed 3 months ago by guilherme.soares

  • Owner changed from guilherme.soares to carlosadean
  • Status changed from new to assigned

comment:2 Changed 3 months ago by carlosadean

  • Status changed from assigned to accepted
  • Component changed from Other to Operation/DataManagement

Olá,

Acho que este dataset nunca esteve no hadoop, mas sim o Y1A1_COADD_SN_D04. Temos registros de processos usando ele de input?

[uhadoop@devel2 ~]$ hdfs dfs -ls /portal_archive/staging/DES/Y1A1_COADD
18/04/03 11:15:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Found 4 items
drwxr-xr-x   - uhadoop des-brazil          0 2016-10-11 17:00 /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_COSMOS_D04
drwxr-xr-x   - uhadoop des-brazil          0 2016-11-08 15:03 /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_SN_D04
drwxr-xr-x   - uhadoop des-brazil          0 2016-11-30 14:02 /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_STRIPE82
drwxr-xr-x   - uhadoop des-brazil          0 2016-11-30 14:02 /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_VVDS14_D04

--
Carlos Adean

comment:3 Changed 3 months ago by julia@…

Oi Carlos,

quanto tempo levaria para fazer a instalação?

É trivial?

são apenas 70 e poucas tiles.


De: "LIneA Ticket System" <ticket@…>
Enviadas: Terça-feira, 3 de abril de 2018 11:16:55
Assunto: Re: [linea] #10446: Y1A1 SN DFULL faltando no hadoop em testing

#10446: Y1A1 SN DFULL faltando no hadoop em testing


Reporter: julia@… | Owner: carlosadean

Type: task | Status: accepted

Priority: normal | Milestone: no milestone

Component: | Severity: normal

Operation/DataManagement? | Keywords:

Resolution: | Add Hours to Ticket: 0

Estimated Number of Hours: 0.0 | Total Hours: 0

Deviation?: 1 |


Changes (by carlosadean):

Comment:

Olá,

Acho que este dataset nunca esteve no hadoop, mas sim o Y1A1_COADD_SN_D04.
Temos registros de processos usando ele de input?

[uhadoop@devel2 ~]$ hdfs dfs -ls /portal_archive/staging/DES/Y1A1_COADD
18/04/03 11:15:05 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
Found 4 items
drwxr-xr-x   - uhadoop des-brazil          0 2016-10-11 17:00
/portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_COSMOS_D04
drwxr-xr-x   - uhadoop des-brazil          0 2016-11-08 15:03
/portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_SN_D04
drwxr-xr-x   - uhadoop des-brazil          0 2016-11-30 14:02
/portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_STRIPE82
drwxr-xr-x   - uhadoop des-brazil          0 2016-11-30 14:02
/portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_VVDS14_D04

--
Carlos Adean

--
Ticket URL: <http://ticket.linea.gov.br/ticket/10446#comment:2>
linea <http://ticket.linea.gov.br>
Sistema de Ticket do LIneA

comment:4 Changed 3 months ago by carlosadean@…

A 'versão' databse já está registrada, certo? Se sim então só bastaria ingerir essa do hadoop.

--
Carlos Adean
IT Team
linea.gov.br
skype: carlosadean


De: "LIneA Ticket System" <ticket@…>
Enviadas: Terça-feira, 3 de abril de 2018 11:21:27
Assunto: Re: [linea] #10446: Y1A1 SN DFULL faltando no hadoop em testing

#10446: Y1A1 SN DFULL faltando no hadoop em testing


Reporter: julia@… | Owner: carlosadean

Type: task | Status: accepted

Priority: normal | Milestone: no milestone

Component: | Severity: normal

Operation/DataManagement? | Keywords:

Resolution: | Add Hours to Ticket: 0

Estimated Number of Hours: 0.0 | Total Hours: 0

Deviation?: 1 |


Comment (by julia@…):

Oi Carlos,

quanto tempo levaria para fazer a instalação?

É trivial?

são apenas 70 e poucas tiles.

----- Mensagem original -----
De: "LIneA Ticket System" <ticket@…>
Enviadas: Terça-feira, 3 de abril de 2018 11:16:55
Assunto: Re: [linea] #10446: Y1A1 SN DFULL faltando no hadoop em testing

#10446: Y1A1 SN DFULL faltando no hadoop em testing
-------------------------------------+-------------------------------------

Reporter: julia@… | Owner: carlosadean

Type: task | Status: accepted

Priority: normal | Milestone: no milestone

Component: | Severity: normal

Operation/DataManagement? | Keywords:

Resolution: | Add Hours to Ticket: 0

Estimated Number of Hours: 0.0 | Total Hours: 0

Deviation?: 1 |

-------------------------------------+-------------------------------------
Changes (by carlosadean):

Comment:

Olá,

Acho que este dataset nunca esteve no hadoop, mas sim o

Y1A1_COADD_SN_D04.

Temos registros de processos usando ele de input?

  [uhadoop@devel2 ~]$ hdfs dfs -ls /portal_archive/staging/DES/Y1A1_COADD
  18/04/03 11:15:05 WARN util.NativeCodeLoader: Unable to load native-
 hadoop
  library for your platform... using builtin-java classes where applicable
  Found 4 items
  drwxr-xr-x   - uhadoop des-brazil          0 2016-10-11 17:00
  /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_COSMOS_D04
  drwxr-xr-x   - uhadoop des-brazil          0 2016-11-08 15:03
  /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_SN_D04
  drwxr-xr-x   - uhadoop des-brazil          0 2016-11-30 14:02
  /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_STRIPE82
  drwxr-xr-x   - uhadoop des-brazil          0 2016-11-30 14:02
  /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_VVDS14_D04

--
Carlos Adean

--
Ticket URL: <http://ticket.linea.gov.br/ticket/10446#comment:2>
linea <http://ticket.linea.gov.br>
Sistema de Ticket do LIneA

--
Ticket URL: <http://ticket.linea.gov.br/ticket/10446#comment:3>
linea <http://ticket.linea.gov.br>
Sistema de Ticket do LIneA

comment:5 Changed 3 months ago by julia@…

Certo. A 'versão' databse já está registrada.


De: "LIneA Ticket System" <ticket@…>
Enviadas: Terça-feira, 3 de abril de 2018 12:50:23
Assunto: Re: [linea] #10446: Y1A1 SN DFULL faltando no hadoop em testing

#10446: Y1A1 SN DFULL faltando no hadoop em testing


Reporter: julia@… | Owner: carlosadean

Type: task | Status: accepted

Priority: normal | Milestone: no milestone

Component: | Severity: normal

Operation/DataManagement? | Keywords:

Resolution: | Add Hours to Ticket: 0

Estimated Number of Hours: 0.0 | Total Hours: 0

Deviation?: 1 |


Comment (by carlosadean@…):

A 'versão' databse já está registrada, certo? Se sim então só bastaria
ingerir essa do hadoop.

--
Carlos Adean
IT Team
linea.gov.br
skype: carlosadean

----- Mensagem original -----
De: "LIneA Ticket System" <ticket@…>
Enviadas: Terça-feira, 3 de abril de 2018 11:21:27
Assunto: Re: [linea] #10446: Y1A1 SN DFULL faltando no hadoop em testing

#10446: Y1A1 SN DFULL faltando no hadoop em testing
-------------------------------------+-------------------------------------

Reporter: julia@… | Owner: carlosadean

Type: task | Status: accepted

Priority: normal | Milestone: no milestone

Component: | Severity: normal

Operation/DataManagement? | Keywords:

Resolution: | Add Hours to Ticket: 0

Estimated Number of Hours: 0.0 | Total Hours: 0

Deviation?: 1 |

-------------------------------------+-------------------------------------

Comment (by julia@…):

Oi Carlos,

quanto tempo levaria para fazer a instalação?

É trivial?

são apenas 70 e poucas tiles.

----- Mensagem original -----
De: "LIneA Ticket System" <ticket@…>
Enviadas: Terça-feira, 3 de abril de 2018 11:16:55
Assunto: Re: [linea] #10446: Y1A1 SN DFULL faltando no hadoop em testing

#10446: Y1A1 SN DFULL faltando no hadoop em testing

-------------------------------------+-------------------------------------

Reporter: julia@… | Owner: carlosadean

Type: task | Status: accepted

Priority: normal | Milestone: no

milestone

Component: | Severity: normal

Operation/DataManagement? | Keywords:

Resolution: | Add Hours to Ticket: 0

Estimated Number of Hours: 0.0 | Total Hours: 0

Deviation?: 1 |

-------------------------------------+-------------------------------------

Changes (by carlosadean):

Comment:

Olá,

Acho que este dataset nunca esteve no hadoop, mas sim o

Y1A1_COADD_SN_D04.

Temos registros de processos usando ele de input?

   [uhadoop@devel2 ~]$ hdfs dfs -ls /portal_archive/staging/DES/Y1A1_COADD
   18/04/03 11:15:05 WARN util.NativeCodeLoader: Unable to load native-
  hadoop
   library for your platform... using builtin-java classes where applicable
   Found 4 items
   drwxr-xr-x   - uhadoop des-brazil          0 2016-10-11 17:00
   /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_COSMOS_D04
   drwxr-xr-x   - uhadoop des-brazil          0 2016-11-08 15:03
   /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_SN_D04
   drwxr-xr-x   - uhadoop des-brazil          0 2016-11-30 14:02
   /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_STRIPE82
   drwxr-xr-x   - uhadoop des-brazil          0 2016-11-30 14:02
   /portal_archive/staging/DES/Y1A1_COADD/Y1A1_COADD_VVDS14_D04

--
Carlos Adean

--
Ticket URL: <http://ticket.linea.gov.br/ticket/10446#comment:2>
linea <http://ticket.linea.gov.br>
Sistema de Ticket do LIneA

--
Ticket URL: <http://ticket.linea.gov.br/ticket/10446#comment:3>
linea <http://ticket.linea.gov.br>
Sistema de Ticket do LIneA

--
Ticket URL: <http://ticket.linea.gov.br/ticket/10446#comment:4>
linea <http://ticket.linea.gov.br>
Sistema de Ticket do LIneA

comment:6 Changed 3 months ago by carlosadean

  • Status changed from accepted to closed
  • Resolution set to fixed

Finalizei a ingestão destes dados no hadoop em testing.

Qualquer problema por favor reabra o ticket.

--
Carlos Adean

Note: See TracTickets for help on using tickets.