Merge staging into production, 14 Aug 2024 #1601
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
adds blocks for title and scripts so that templates can put those parts in the header.
The title and scripts are no longer simply added to the body of the page, but are now actually contained by the header, where they should be.
Since the siglum and institution name now come from the institution, the source only needs to have the shelfmark. However, until all the data is properly migrated, the old title and siglum fields will remain.
This is a hacky, change-as-needed script that serves to migrate the source data from the old structure to the new, creating institutions and moving the shelfmark data to the new sources. There is a whole bunch of sigla that get skipped (essentially all private collectors) until we figure out how they will be handled. Don't look to closely, since there are lots of skeletons in this closet.
Allows sources to be added to an institution by using the autocomplete widget
Previously, the LoginView import in the URLs was taken from the views.user module, instead of directly from the django auth views, because this view was also imported into the user module. (aka, a transitive import). This changes it to import from the auth views directly. I also ran a formatter on the imports in the user module, so that's why it looks like there are more changes than there actually are!
Previously there were several duplicated property definitions that varied only by checking whether the user was logged in. They also did the sorting in Python, rather than in the database. This condenses all the definitions to a single one per property, and uses the expanding empty dict trick to determine whether `published=True` gets added to the filters. (Since without this, what is really desired is `published = True OR False` but without resorting to Q queries this was the simplest way to do it.)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.2.1 to 2.2.2. - [Release notes](https://github.com/urllib3/urllib3/releases) - [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst) - [Commits](urllib3/urllib3@2.2.1...2.2.2) --- updated-dependencies: - dependency-name: urllib3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]>
This commit completely reworks the fetching of data for the feast detail page. Previously, this page took several seconds to load, and generated a large number of queries, due to the complexity of the information needing to be collated. This was rewritten to instead use two custom SQL queries that are executed directly. This means that the number of queries generated by this segment of code is now no more than two. There are a few 'interesting' things in this, which I've tried to note in comments. In the process, this also fixes a number of other small bugs: Fixes #1540 Fixes #1136 Fixes #596
This commit reworks the source display in the templates, primarily by calling a couple helper methods on the Source model to keep the referencing consistent. The 'short heading' method displays the Institution siglum and the shelfmark, while the 'heading' displays the full institution name and the shelfmark. The edit and create templates were reworked for the new structure and source referencing. The source display will now show the institution and the shelfmark in separate columns.
This is to allow saving sources in the create / edit without triggering a validation error.
Buckle up, this one's a bit messy. This commit disables the Feast Detail tests. It turns out, after much searching and Googling and digging, that the Django Test Suite always wraps the tests in a transaction, so that the test can be easily rolled back after each one is run, and thus present a database in a known state for each individual test. However, with the switch to running a raw SQL query, the cursor in the Feast Detail View now runs outside of this transaction. Thus to the View, the database always looks empty, even if it's running inside of the test. I tried fixtures, TransactionTestCase, overriding methods and hacking the TestCase to prevent this behaviour, but no dice. So for now I've marked the Feast Detail tests as "skip" so that they still appear in the runs, but are skipped because they will always fail, as they're written.
It's better to use the named fields for the column content than to rely on the order of the tuple unpacking....
Still failing, but being worked on. Committing here to merge in latest changes.
# Conflicts: # django/cantusdb_project/main_app/templates/browse_chants.html # django/cantusdb_project/main_app/templates/chant_create.html # django/cantusdb_project/main_app/templates/chant_detail.html # django/cantusdb_project/main_app/templates/chant_edit.html # django/cantusdb_project/main_app/templates/melody_search.html # django/cantusdb_project/main_app/templates/source_detail.html # django/cantusdb_project/main_app/templates/source_edit.html # django/cantusdb_project/main_app/templates/user_source_list.html
Syncing the templates with the latest develop generated a number of merge conflicts. This commit fixes them. Also it fixes the new block definitions in the base templates. Fixes #1551
The numbers from a fresh database copy now match the numbers running on the production site, and the numbers fetched in the browse Chants UI for the chants associated with a feast.
Adds a section that shows a list of sources that are attached to the institution at the bottom of the institution page. Also adds the ID as a link field to that source record.
This commit adds the ability to specify private collectors as a type of Institution. The model constraints are set up so that you can either specify that an institution is a private collector (via a checkbox) OR they have a siglum. Both cannot be selected for the same record, but one of the two options must be selected for each record. Institutions that do not have a siglum currently get the word "Private" prepended to the shelfmark. All other fields are accessible for private collectors. Alters the admin UI to make the distinction via the layout as well.
Puts the siglum at the front, so that the fact that it is ordered by siglum is clear.
Use select_related to reduce queries on json_cid_export view
fix(api): optimize json-cid view queries
Handle errors with improper chant text field characters/syllabification fails
Merge `develop` into `staging`, 11 Oct 2024
Adjust options for Source.source_completeness field to "Full source", "Fragment", and "Reconstruction". Adjust display for Source.complete_inventory boolean to `True` = "Full inventory" and `False` = "Partial inventory".
…ompleteness = "Full Source" Sources where `full_source` was `None` are now mapped by the `populate_source_completeness` command to `source_completeness` = "Full source". Updates `make_fake_source` to use `source_completeness` field. Updates test of `populate_source_completeness` field.
Change hardcoded name for Bower segment sources. This will be modified as part of our segment overhaul (see #1668)
Adds a template tag `join_absolute_url_links` to create a list of links to object detail pages. Uses this tag to display multiple centuries when a source has multiple centuries on the source list and source detail pages.
Fix source_completeness field display and filtering behaviour
Merge `develop` into `staging`, 18 October 2024
|
@lucasmarchd01 Ok, imho this is ready to go... thoughts? |
|
Sounds good to me. Exciting that we're finally getting these changes up, thanks for all your hard work! |
|
Thank you! And thanks @ahankinson! I'll start shortly. |
dchiller
approved these changes
Oct 21, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Global Changes
Source/Institution Identification
This was implemented in many PRs (#1545, #1566, #1568, #1570, #1584, #1586, #1645, #1615, #1606, #1629, #1618). There are many conversations and emails for all of the different pieces of this transition (too many to really summarize here). However, the key points are:
Source,SourceIdentifier,Institution, andInstitutionIdentifiermigrations run correctlymigrate_recordscommand runs correctly onFrontend/Templates
Source Views
datefield inSourcemodel. Black formatting changes by @lucasmarchd01 in Change help text fordatefield inSourcemodel. Black formatting changes #1567datefield on source edit page by @lucasmarchd01 in Change help text fordatefield on source edit page #1603MelodySearchView#1648)source_completenessfield options display as "Complete Source", "Fragment", "Reconstruction" andcomplete_inventoryfield options display as "Full inventory" and "partial inventory" (Fix source_completeness field display and filtering behaviour #1666)Chant Views
Chant Search
Commands
remap_user_idsworks (Commands: fixremap_user_idsand addmap_cantus_ids#1662)map_cantus_idsworks (Commands: fixremap_user_idsand addmap_cantus_ids#1662)reformat_source_idsworks (Add command to standardize formatting of DACT IDs and Fragmentarium IDs #1651)populate_source_completenessworks (Adjust source fields: holding_institution, name, production_method, source_completeness #1645)full_source = Nonemap tosource_completeness = "Complete Source"(Fix source_completeness field display and filtering behaviour #1666)APIs
json-cidendpoint works with 909000 (fix(api): optimize json-cid view queries #1656)json-cid/909000leads to timeout #1655)ServiceandGenrelist and detail views (Create JSON endpoint forServiceandGenrelist and detail views #1612)Admin
FieldErroron admin user change page (FixFieldErroron admin user change page #1600)Code cleanliness/refactoring
test_viewsto match separate view class files (various)Commands to run on production (copying from #1663):
0. Run migrations
Two sources (699079 and 698806) have been "pre-migrated" and will need manual attention when production is updated.
Printed sources will need manual attention after updating:
production_methodfield