Aprimora fluxo de processamento de artigos e faz correcoes journal issue etc

[robertatakenaka]

Descrição

Refatoração do pipeline de processamento de artigos, consolidando múltiplas tasks Celery em duas tasks centrais (task_dispatch_articles e task_process_article_pipeline), introduzindo o modelo XMLURL para rastreamento de falhas de coleta, e corrigindo uma série de problemas menores em modelos, índices de busca e agendamento de tarefas.

---

Motivação

O pipeline anterior era composto por dezenas de tasks Celery com responsabilidades sobrepostas (task_select_articles_to_complete_data, task_load_articles, task_load_journal_articles, task_select_articles_to_load_from_api, task_select_articles_to_load_from_collection_endpoint, task_load_article_from_xml_url, task_load_article_from_pp_xml, entre outras), dificultando manutenção, rastreabilidade de erros e reprocessamento seletivo. Além disso, falhas de rede ao coletar XMLs ficavam registradas apenas em UnexpectedEvent, sem estrutura para reprocessamento ou análise de padrões.

---

Principais mudanças

article/controller.py — ArticleIteratorBuilder

Nova classe que encadeia até quatro iteradores de seleção de artigos:

• _iter_from_pid_provider: filtra PidProviderXML por ISSN, ano e proc_status_list
• _iter_from_article: filtra Article por data_status; tenta recuperar pp_xml via get_by_pid_v3; emite None para artigos sem pp_xml (sinaliza skip)
• _iter_from_harvest: coleta via OPACHarvester (scl) ou AMHarvester (demais coleções)
• _iter_from_article_source: filtra ArticleSource por status e datas

article/tasks.py — consolidação de tasks

Tasks removidas:

• task_select_articles_to_complete_data
• task_select_articles_to_load_from_api
• task_select_articles_to_load_from_collection_endpoint
• task_load_article_from_xml_url
• task_load_article_from_pp_xml
• task_select_articles_to_load_from_article_source
• task_load_articles
• task_load_journal_articles
• task_fix_journal_articles_status (lógica absorvida por task_fix_article_status)

Tasks novas:

• task_dispatch_articles: orquestradora que instancia ArticleIteratorBuilder e dispara task_process_article_pipeline.delay para cada item
• task_process_article_pipeline: pipeline com três pontos de entrada:
  • Fluxo A: xml_url + collection_acron + pid → AMArticle → ArticleSource → add_pid_provider → load_article
  • Fluxo B: article_source_id → add_pid_provider → load_article
  • Fluxo C: pp_xml_id direto → load_article

article/models.py — ArticleSource

• Renomeia complete_data → add_pid_provider com pipeline em duas etapas (request_xml + request_pid), com skip inteligente quando arquivo/pid já existem e force_update=False
• Renomeia get_or_create_pid_v3 → request_pid; usa get_by_pid_v3; lança UnableToRegisterPIDError
• Novos StatusChoices: URL_ERROR e XML_ERROR
• Novas exceções tipadas: RequestXMLException, XMLException, UnableToRegisterPIDError
• Novo cached_property xml_with_pre
• is_completed: logging detalhado em cada condição de retorno False
• ArticleAffiliation: adiciona autocomplete_label e autocomplete_custom_queryset_filter

pid_provider/models.py — XMLURL e refatorações

Novo modelo XMLURL:
Rastreia todas as URLs processadas pelo PidProvider com status, pid, zipfile (ZIP comprimido em memória) e exceptions (máx. 255 chars).

PidProviderXML:

• Novo get_by_pid_v3(pid_v3, partial_pid_v2, pid_v2): busca flexível; em MultipleObjectsReturned retorna o mais recente
• find_duplicated_pkg_names: retorna list(set(...)) em vez de values_list flat
• Remove find_duplicated_v2 (não utilizado)
• Novo mark_items_as_duplicated(issns): bulk update
• deduplicate_items: simplificado, delega para fix_duplicated_pkg_name
• Renomeia fix_duplicated_items → fix_duplicated_pkg_name; remove lógica por v2
• Novo método de instância fix_pkg_name(pkg_name)

XMLVersion:

• get_or_create: trata AttributeError/TypeError/ValueError ao verificar arquivo em disco; usa pid_provider_xml.v3 como nome de arquivo

pid_provider/base_pid_provider.py

Refatora provide_pid_for_xml_uri com três categorias de erro:

• a) Falha ao obter XML → registra XMLURL com xml_fetch_failed
• b) XML obtido, registro falhou → registra XMLURL com pid_provider_xml_failed + salva zipfile
• c) Erro inesperado → registra em UnexpectedEvent

collection/models.py

Adiciona cached_property base_url: retorna domain com https:// prefixado quando ausente.

article/search_indexes.py

ArticleIndex — novos campos: issn, license, aff_country, aff_institution, open_access, indexed_at, crossmark_active

ArticleOAIIndex — novos campos: issn, publisher, orcid, format_; prepare_date simplificado para retornar obj.pub_date

Correção de URLs: substitui 'http://%s' % collection.domain por '%s' % collection.base_url em todos os campos de URL dos dois índices.

Outras mudanças

• core/views.py: absorve CommonControlFieldViewSet (antes em core/viewsets.py, arquivo removido)
• issue/tasks.py: renomeia load_issue_from_article_meta → load_issue_from_articlemeta
• journal/choices.py: remove definição duplicada de UPDATE_POLICY_TYPE
• journal/models.py: remove classname='collapsed' de todos os InlinePanel
• journal/wagtail_hooks.py: adiciona updated ao list_display de 7 admins
• bigbang/tasks_scheduler.py: registra todas as tasks obsoletas para limpeza; agenda task_dispatch_articles no lugar das tasks removidas
• locale/es: Publisher → Entidad Editora
• requirements/base.txt: packtools 4.15.0 → 4.16.1

---

Migrações

App	Migração	Descrição
article	0048_alter_articlesource_status	Adiciona url_error e xml_error ao campo status
pid_provider	0015_alter_xmlversion_file_xmlurl	Amplia max_length de XMLVersion.file; cria modelo XMLURL

---

Testes

Adicionados em pid_provider/test_models.py:

• XMLURLTest (8 casos): criação, recuperação, atualização, save_file com string/bytes/filename padrão, __str__
• BasePidProviderXMLURITest (3 casos): falha de fetch, sucesso, falha de registro

python manage.py test pid_provider.test_models

---

Checklist

• Migrações criadas e revisadas
• Tasks obsoletas adicionadas à lista de limpeza do tasks_scheduler
• Testes adicionados para XMLURL e BasePidProvider
• Docstrings adicionadas/atualizadas nas tasks
• Imports não utilizados removidos
• Validar em ambiente de homologação com coleção real (scl)
• Confirmar que tasks obsoletas não estão agendadas no Celery Beat antes do deploy

[Copilot]

Pull request overview

Esta PR refatora o pipeline de processamento de artigos (reduzindo várias tasks antigas para duas tasks centrais) e adiciona rastreabilidade de falhas de coleta/registro via o novo modelo XMLURL, além de ajustes em admin, índices de busca e agendamentos.

Changes:

• Consolida o processamento em task_dispatch_articles + task_process_article_pipeline e introduz ArticleIteratorBuilder para seleção de itens de múltiplas fontes.
• Adiciona o modelo pid_provider.XMLURL e integra seu uso no BasePidProvider para registrar falhas/sucessos e persistir o XML compactado.
• Atualiza índices de busca e pequenos ajustes em admin/migrações/traduções/scheduler.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
pid_provider/wagtail_hooks.py	Atualiza import do CommonControlFieldViewSet após mudança de localização.
pid_provider/test_models.py	Adiciona testes para XMLURL e cenários de BasePidProvider.
pid_provider/sources/harvesting.py	Usa PidProviderXML.get_by_pid_v3() para reaproveitar registros quando possível.
pid_provider/models.py	Ajusta XMLVersion.file e adiciona o novo modelo XMLURL e helper get_by_pid_v3.
pid_provider/migrations/0015_alter_xmlversion_file_xmlurl.py	Migração para XMLVersion.file(max_length=300) e criação do modelo XMLURL.
pid_provider/base_pid_provider.py	Registra falhas/sucessos em XMLURL e mantém UnexpectedEvent para erros inesperados.
organization/migrations/0012_alter_organization_url.py	Ajuste de URLField para permitir null/blank.
locale/es/LC_MESSAGES/django.po	Ajuste de tradução de “Publisher”.
journal/wagtail_hooks.py	Inclui updated no list_display em múltiplos admins.
journal/models.py	Remove classname="collapsed" de InlinePanels.
journal/migrations/0059_alter_digitalpreservationagency_options.py	Ajusta verbose_name/verbose_name_plural.
journal/choices.py	(Re)declara UPDATE_POLICY_TYPE.
issue/tasks.py	Renomeia task load_issue_from_article_meta → load_issue_from_articlemeta.
institution/migrations/0008_alter_institution_url_alter_scimago_url.py	Ajuste de URLField para permitir null/blank.
core/views.py	Adiciona CommonControlFieldViewSet baseado em SnippetViewSet com save_all(user) quando disponível.
core/home/migrations/0015_alter_formpage_thank_you_text.py	Ajuste de campo RichTextField (help_text/blank).
collection/models.py	Adiciona base_url para normalizar domínio com protocolo.
bigbang/tasks_scheduler.py	Limpa tasks obsoletas e agenda task_dispatch_articles no lugar das removidas.
article/tasks.py	Implementa task_dispatch_articles e task_process_article_pipeline e remove/absorve tasks antigas.
article/sources/xmlsps.py	Permite obter pp_xml via PidProviderXML.get_by_pid_v3 quando só há v3.
article/search_indexes.py	Novos campos e correção de URLs via collection.base_url; simplifica prepare_dates.
article/models.py	Refatora ArticleSource para add_pid_provider (request_xml + request_pid) e novos status/erros.
article/migrations/0048_alter_articlesource_status.py	Adiciona url_error e xml_error em ArticleSource.status.
article/controller.py	Introduz ArticleIteratorBuilder para encadear fontes de seleção/dispatch.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The error message mentions pid_v3 as a valid entry point, but this task doesn’t accept a pid_v3 parameter. This is misleading for operators troubleshooting task failures; update the message to list only supported inputs (xml_url, article_source_id, pp_xml_id).

Suggested change

	"article_source_id, pp_xml_id or pid_v3."
	"article_source_id, or pp_xml_id."

[Copilot]

PidProviderXML.get_by_pid_v3 builds a params dict and passes it to objects.get(**params). When both pid_v3 and pid_v2 are provided this becomes an AND filter, which is not a “flexible” lookup and can fail even when either PID matches. Also, if all inputs are falsy, params is empty and get() can raise MultipleObjectsReturned (or return an arbitrary single row), which is unsafe. Consider guarding against empty input and using a Q-based OR lookup across v3/v2/partial v2, returning the most recent match deterministically.

Suggested change

	params = {}
	if pid_v3:
	params["v3"] = pid_v3
	if pid_v2:
	params["v2"] = pid_v2
	if partial_pid_v2:
	params["v2__contains"] = partial_pid_v2
	try:
	return cls.objects.get(**params)
	except cls.MultipleObjectsReturned as e:
	return cls.objects.filter(**params).order_by("-updated").first()
	"""
	Retrieve a PidProviderXML instance by one or more PID values.
	The lookup is flexible:
	- Matches by v3 when ``pid_v3`` is provided.
	- Matches by v2 when ``pid_v2`` is provided.
	- Matches by partial v2 (substring) when ``partial_pid_v2`` is provided.
	When multiple values are provided, results are combined with OR and
	the most recently updated matching record is returned.
	"""
	# Guard against an empty lookup which would otherwise result in an
	# unfiltered query that may return arbitrary or multiple rows.
	if not (pid_v3 or pid_v2 or partial_pid_v2):
	raise cls.DoesNotExist(
	"PidProviderXML.get_by_pid_v3 called without any PID argument"
	)
	query_parts = []
	if pid_v3:
	query_parts.append(Q(v3=pid_v3))
	if pid_v2:
	query_parts.append(Q(v2=pid_v2))
	if partial_pid_v2:
	query_parts.append(Q(v2__contains=partial_pid_v2))
	# Combine all query parts using OR logic.
	combined_query = query_parts[0]
	for extra_q in query_parts[1:]:
	combined_query |= extra_q
	# Deterministically return the most recently updated matching record.
	obj = cls.objects.filter(combined_query).order_by("-updated").first()
	if obj is None:
	raise cls.DoesNotExist(
	f"No PidProviderXML found for the provided PID arguments: "
	f"pid_v3={pid_v3!r}, pid_v2={pid_v2!r}, partial_pid_v2={partial_pid_v2!r}"
	)
	return obj

[Copilot]

PidProviderXML.deduplicate_items had its signature changed to accept only (user, issns), but there are still call sites passing mark_as_duplicated / deduplicate keyword args (e.g. pid_provider/tasks.py:285). This will raise TypeError at runtime. Either keep backwards-compatible kwargs or update all callers accordingly.

[Copilot]

_handle_pid_provider_failure reads response.get("error_message"), but PidProviderXML.register() (and other codepaths) populate error_msg instead. This will often record "Unknown error" in XMLURL.exceptions even when an error message exists. Consider falling back to error_msg when error_message is absent.

Suggested change

	error_msg = response.get("error_message", "Unknown error")
	error_msg = response.get("error_message") or response.get("error_msg") or "Unknown error"

[Copilot]

UPDATE_POLICY_TYPE is defined twice in this module, which can lead to confusion and accidental edits to the wrong copy (the latter definition silently overrides the former). Remove the duplicate and keep a single canonical definition.

Suggested change

	UPDATE_POLICY_TYPE = [
	("correction", _("Correction")),
	("retraction", _("Retraction")),
	("partial-retraction", _("Partial Retraction")),
	("withdrawal", _("Withdrawal")),
	("expression-of-concern", _("Expression of Concern")),
	("other", _("Other")),
	]

[Copilot]

ArticleIteratorBuilder.__iter__ unconditionally yields from all four iterators. This contradicts the class docstring (which says iterators are activated by “exclusive” args) and means every dispatch will also harvest all collections (and also iterate PidProviderXML + Article + ArticleSource), potentially duplicating work and massively increasing load. Make iterator selection conditional so only the requested sources run (and default to only _iter_from_pid_provider when no exclusive args are provided).

Suggested change

	yield from self._iter_from_harvest()
	yield from self._iter_from_article_source()
	yield from self._iter_from_pid_provider()
	yield from self._iter_from_article()
	"""
	Itera sobre artigos de acordo com os argumentos "exclusivos".
	A seleção dos iteradores segue a documentação da classe:
	- Se `opac_url` foi informado, itera a partir do harvest
	(`_iter_from_harvest`).
	- Se `article_source_status_list` foi informado, itera a partir de
	`ArticleSource` (`_iter_from_article_source`).
	- Se `data_status_list` foi informado, itera a partir de `Article`
	(`_iter_from_article`).
	- Se `proc_status_list` foi informado, itera a partir de
	`PidProviderXML` (`_iter_from_pid_provider`).
	Caso nenhum desses argumentos "exclusivos" seja informado, o
	comportamento padrão é iterar apenas a partir de `PidProviderXML`.
	"""
	any_selected = False
	# Harvest somente quando explicitamente solicitado (por exemplo, via opac_url)
	if self.opac_url is not None:
	any_selected = True
	yield from self._iter_from_harvest()
	# ArticleSource controlado por article_source_status_list
	if self.article_source_status_list is not None:
	any_selected = True
	yield from self._iter_from_article_source()
	# Article controlado por data_status_list
	if self.data_status_list is not None:
	any_selected = True
	yield from self._iter_from_article()
	# PidProviderXML controlado por proc_status_list
	if self.proc_status_list is not None:
	any_selected = True
	yield from self._iter_from_pid_provider()
	# Padrão: apenas PidProviderXML quando nenhum argumento exclusivo foi informado
	if not any_selected:
	yield from self._iter_from_pid_provider()

[Copilot]

In the xml_url flow, ArticleSource.create_or_update() triggers add_pid_provider(), which catches errors internally and may leave article_source.pid_provider_xml unset. Accessing article_source.pid_provider_xml.id will then raise AttributeError and turn expected pipeline failures into UnexpectedEvent noise. Consider checking article_source.status/pid_provider_xml and returning early (or raising a clearer error) when PID registration fails.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 11 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The schema adds indexed_at, but the updated ArticleIndex now sends additional fields (e.g., issn, aff_country, aff_institution, crossmark_active). For multi-valued fields, Solr needs multiValued="true" definitions; otherwise they’ll fall back to the * dynamicField (multiValued="false") and indexing will fail when Haystack submits lists/booleans. Add explicit <field ...> entries (with correct types/multiValued) for each new indexed field introduced in article/search_indexes.py.

Suggested change

	<!-- Additional fields indexed by ArticleIndex -->
	<field name="issn" type="string" indexed="true" stored="true" multiValued="true"/>
	<field name="aff_country" type="string" indexed="true" stored="true" multiValued="true"/>
	<field name="aff_institution" type="string" indexed="true" stored="true" multiValued="true"/>
	<field name="crossmark_active" type="boolean" indexed="true" stored="true" multiValued="false"/>

[Copilot]

task_dispatch_articles logs every dispatched item at INFO (Dispatching article with kwargs: ...). For large backfills this can generate very high log volume and impact observability/ingestion costs. Consider downgrading to DEBUG or logging periodic summaries/counters instead.

Suggested change

	logging.info(f"Dispatching article with kwargs: {item_kwargs}")
	logging.debug(f"Dispatching article with kwargs: {item_kwargs}")

[Copilot]

OPACHarvester now expects domain to include the scheme (default https://...) and concatenates URLs with f"{self.domain}/...". The docstring still suggests passing just www.scielo.br, and existing callers still do so. To prevent invalid URLs, normalize domain in __init__ (prepend https:// when missing) or revert to building URLs with an explicit scheme.

[Copilot]

XMLURL.url is created without unique=True/UniqueConstraint, but application code uses XMLURL.get(url=...)/create_or_update() assuming a single row per URL. Consider enforcing uniqueness at the DB level here (or updating code to handle multiple rows) to prevent MultipleObjectsReturned in production.

[Copilot]

XMLURL.create()/create_or_update() assume url is unique (they call get(url=...)), but the model/migration do not enforce uniqueness. This can lead to MultipleObjectsReturned during error handling (e.g., in BasePidProvider._handle_xml_fetch_failure) and make retries unreliable. Either add a DB-level unique constraint on url (and update the migration) or change get()/update logic to tolerate multiple rows (e.g., update the latest record).

[Copilot]

ArticleIteratorBuilder._build_harvester() passes "www.scielo.br" to OPACHarvester, but OPACHarvester now expects domain to already include the scheme (it concatenates with f"{self.domain}/..."). This will generate invalid URLs like www.scielo.br/api/.... Pass a fully qualified base URL (e.g., https://www.scielo.br) or normalize inside OPACHarvester.

Suggested change

	return OPACHarvester(self.opac_url or "www.scielo.br", collection_acron, **kwargs)
	domain = self.opac_url or "https://www.scielo.br"
	if not domain.startswith(("http://", "https://")):
	domain = f"https://{domain}"
	return OPACHarvester(domain, collection_acron, **kwargs)

[Copilot]

aff_country and aff_institution fields were added to ArticleIndex, but there are no corresponding prepare_aff_country / prepare_aff_institution methods. As a result these fields will not be populated (defeating their purpose for filtering). Implement the prepare methods (e.g., aggregating countries/institutions from affiliations) or remove the fields until implemented.

[Copilot]

PidProviderXML.get_by_pid_v3() builds params dynamically, but if callers pass no PID inputs (all params None/empty), it will call objects.get(**{}), which can raise MultipleObjectsReturned unexpectedly. Consider validating that at least one of pid_v3, pid_v2, or partial_pid_v2 was provided and returning None or raising ValueError otherwise.

[Copilot]

provide_pid_for_xml_uri is checking error_message, but PidProviderXML.register() populates error_msg on failures. As-is, failures returned as {error_type, error_msg} will be treated as success and recorded in XMLURL with status success (and _handle_pid_provider_failure will also lose the real message). Update the condition and _handle_pid_provider_failure to consistently use the actual error keys returned by register() (and/or normalize the response in register() to one key).

Suggested change

	if response.get("error_type") or response.get("error_message"):
	if response.get("error_type") or response.get("error_msg"):

[Copilot]

Both _iter_from_pid_provider() and _iter_from_article() call .count() on potentially large querysets (and then iterate them), causing extra full COUNT queries that can be expensive in production. Consider removing these .count() calls (or using a cheap counter while iterating) and logging only the iterated count.