Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

💥 Replace BoltDB Storage to Gorm SQL #237

Merged
merged 7 commits into from
Oct 9, 2023

Conversation

0x46616c6b
Copy link
Member

The current implementation using BoltDB has significant performance issues as the Message table grows. The problem lies in the lack of indexing for the ID and Ticker fields, resulting in lengthy loading times and backpressure on the web server.

This pull request completely revamps the storage system using Gorm and introduces an SQL database (SQLite, MySQL, PostgreSQL). By implementing improved indexing, the overall performance is expected to greatly improve. Additionally, this update enables separating the API from the underlying storage (database).

Please note that this pull request introduces a significant breaking change for the ticker.

@codecov
Copy link

codecov bot commented Oct 7, 2023

Codecov Report

Merging #237 (3f77394) into main (925b67a) will increase coverage by 5.15%.
The diff coverage is 90.71%.

@@            Coverage Diff             @@
##             main     #237      +/-   ##
==========================================
+ Coverage   89.05%   94.20%   +5.15%     
==========================================
  Files          37       37              
  Lines        1681     1622      -59     
==========================================
+ Hits         1497     1528      +31     
+ Misses        156       73      -83     
+ Partials       28       21       -7     
Files Coverage Δ
internal/api/api.go 100.00% <100.00%> (ø)
internal/api/feed.go 100.00% <100.00%> (ø)
internal/api/init.go 100.00% <100.00%> (ø)
internal/api/messages.go 100.00% <100.00%> (ø)
internal/api/middleware/ticker/ticker.go 100.00% <100.00%> (ø)
internal/api/response/message.go 100.00% <100.00%> (ø)
internal/api/response/settings.go 100.00% <100.00%> (+100.00%) ⬆️
internal/api/response/ticker.go 100.00% <100.00%> (+100.00%) ⬆️
internal/api/response/timeline.go 100.00% <100.00%> (ø)
internal/api/response/upload.go 100.00% <100.00%> (ø)
... and 13 more

@0x46616c6b 0x46616c6b force-pushed the Replace-BoltDB-Storage-to-Gorm-SQL branch from 5c74e89 to a7403c9 Compare October 7, 2023 20:33
@0x46616c6b 0x46616c6b force-pushed the Replace-BoltDB-Storage-to-Gorm-SQL branch from 3ef7c5b to 3f77394 Compare October 8, 2023 19:48
@0x46616c6b 0x46616c6b merged commit 50eab74 into main Oct 9, 2023
6 checks passed
@0x46616c6b 0x46616c6b deleted the Replace-BoltDB-Storage-to-Gorm-SQL branch October 9, 2023 16:28
@0x46616c6b 0x46616c6b changed the title Replace BoltDB Storage to Gorm SQL 💥 Replace BoltDB Storage to Gorm SQL Oct 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant