Implement Yahoo Finance API for Stock/ETF Price Oracle #9227 #9587

acc-no-longer-active · 2025-03-10T06:30:03Z

Yahoo Finance API Integration

This PR implements a Yahoo Finance API integration to provide price data for stocks and ETFs, addressing issue #9227. This integration enhances Rotki's ability to track and display prices for traditional financial assets alongside crypto assets.

Implementation Overview

The Yahoo Finance integration is implemented as a new price oracle that follows the existing oracle interfaces in Rotki. It provides both current and historical price data for stocks and ETFs. The implementation:

Uses the yfinance Python library for reliable access to Yahoo Finance data
Implements caching to reduce API calls and avoid rate limiting
Provides robust error handling and retry mechanisms for rate limiting
Adds special handling for custom assets of type 'stock' and 'etf'
Extends the asset schema to support Yahoo Finance ticker symbols

Major Files Updated

New Files

rotkehlchen/externalapis/yahoofinance.py
- Implements the YahooFinance class that inherits from ExternalServiceWithApiKeyOptionalDB, HistoricalPriceOracleWithCoinListInterface, and PenalizablePriceOracleMixin
- Provides methods for querying current and historical prices
- Implements ticker symbol extraction and normalization
- Adds direct API access method for more reliable price fetching
- Implements caching to reduce API calls
- Adds special handling for custom assets with stock/ETF types
- Implements rate limiting protection with exponential backoff
rotkehlchen/tests/external_apis/test_yahoofinance.py
- Unit tests for the Yahoo Finance implementation
- Tests ticker extraction, price querying, and error handling
- Mocks API responses to avoid actual API calls during testing
- Tests custom asset price querying with different ticker formats
- Tests rate limiting retry mechanisms
rotkehlchen/tests/external_apis/test_yahoofinance_integration.py
- Integration tests for the Yahoo Finance API
- Tests actual API calls with rate limiting protection
- Verifies the integration with the Rotki system
- Implements test rate limiting to avoid excessive API calls
- Tests direct API requests and response handling

Modified Files

rotkehlchen/assets/asset.py
- Added yahoo_finance field to AssetWithOracles class
- Implemented to_yahoo_finance() method to extract ticker symbols
- Added support for custom assets with stock/etf types
- Enhanced ticker extraction from custom asset notes
rotkehlchen/inquirer.py
- Integrated Yahoo Finance with the price inquirer system
- Added special handling for custom assets of type stock/etf
- Implemented caching for stock/etf prices
- Added _yahoofinance instance to the Inquirer class
- Added _stock_etf_prices dictionary to track stock/ETF prices with timestamps
- Implemented _should_refresh_stock_price method to determine when to refresh prices
- Added force_refresh_stock_etf_prices method to manually refresh all stock/ETF prices
- Enhanced custom asset price handling to work with Yahoo Finance
rotkehlchen/balances/manual.py
- Enhanced add_manually_tracked_balances to handle custom assets of type stock/etf
- Added special price handling for stock/ETF custom assets
- Implemented price extraction from manual balance entries
- Added logging for custom asset price tracking
- Modified price storage mechanism to work with Yahoo Finance
rotkehlchen/tests/unit/test_manual_balances.py
- Added tests for manually tracked balances with custom stock/ETF assets
- Tests price extraction and storage for custom assets
- Tests editing manually tracked balances with custom assets
- Tests price updates for custom assets
rotkehlchen/tests/api/test_custom_assets.py
- Enhanced tests for custom assets to include stock/ETF types
- Tests API endpoints for custom asset creation and modification
- Tests custom asset type validation
rotkehlchen/oracles/structures.py
- Added YAHOOFINANCE to CurrentPriceOracle enum
rotkehlchen/types.py
- Added YAHOO_FINANCE to ExternalService enum
rotkehlchen/rotkehlchen.py
- Initialized Yahoo Finance instance alongside other price oracles
rotkehlchen/history/price.py
- Integrated Yahoo Finance with the price historian
frontend/app/src/locales/en.json
- Added localization for "Yahoo Finance"
frontend/app/src/components/helper/PrioritizedListEntry.vue
- Added Yahoo Finance to the price oracle UI components

Integration Details

Yahoo Finance API Integration

The Yahoo Finance API integration is designed to work seamlessly with Rotki's existing price oracle system. The implementation:

Follows Oracle Interface: Implements the required interfaces (HistoricalPriceOracleWithCoinListInterface, PenalizablePriceOracleMixin) to ensure compatibility with the existing system.
Robust Error Handling: Implements comprehensive error handling with detailed logging to help diagnose issues.
Rate Limiting Protection: Uses exponential backoff for retry logic when rate limiting is encountered.
Caching: Implements an in-memory cache to reduce API calls and improve performance.

Custom Asset Integration

The integration with custom assets is a key feature of this implementation:

Ticker Extraction: Implements multiple methods to extract ticker symbols from custom assets:
- From yahoo_finance attribute
- From notes with "ticker:" prefix
- From asset symbol
- From asset name with common suffixes removed
Price Storage: Enhances the manual balance system to store and retrieve prices for custom stock/ETF assets.
Price Refresh Logic: Implements logic to determine when to refresh stock/ETF prices based on time elapsed.

Inquirer Integration

The integration with the Inquirer system is designed to be non-intrusive:

Special Handling: Adds special handling for custom assets in the price inquirer to ensure Yahoo Finance is used for stock/ETF assets.
Price Caching: Implements a dedicated caching system for stock/ETF prices to reduce API calls.
Background Refresh: Adds background refresh capability for stock/ETF prices to ensure up-to-date data.

Manual Balances Integration

The integration with manual balances enhances the user experience for tracking stock/ETF assets:

Price Extraction: Extracts prices from manual balance entries for custom stock/ETF assets.
Price Storage: Stores prices in the custom asset price system for future use.
Price Updates: Updates prices when manual balances are edited.

Testing

The implementation includes comprehensive tests:

Unit Tests: Tests individual components of the Yahoo Finance implementation:
- Ticker extraction from different asset types
- Price querying for stocks and ETFs
- Error handling and retry mechanisms
- Custom asset price handling
Integration Tests: Tests the integration with the Rotki system:
- Actual API calls with rate limiting protection
- Integration with the price inquirer
- Integration with manual balances
- Integration with custom assets
Rate Limiting Protection: Tests are designed to avoid excessive API calls:
- Uses persistent files to track when tests were last run
- Skips tests if they were run recently
- Adds random delays to avoid synchronized tests
- Skips tests in CI environments
Mock Responses: Uses mock responses for tests that don't need actual API calls.

Future Improvements

Potential future enhancements to the Yahoo Finance integration:

Expanded Asset Support: Add support for more asset types beyond stocks and ETFs.
Symbol Mapping Database: Implement a more comprehensive symbol mapping database for better ticker resolution.
Additional Data Points: Add support for dividends, corporate actions, and other financial data.
UI Enhancements: Improve the UI for displaying stock/ETF information.
Batch Queries: Implement batch queries to reduce API calls further.
Advanced Caching: Implement more sophisticated caching strategies with persistence.
Historical Data: Enhance historical data retrieval for better performance analysis.

Related Issues

Closes #9227

…ustom assets / manual balances

CLAassistant · 2025-03-10T06:30:09Z

All committers have signed the CLA.

acc-no-longer-active · 2025-03-10T06:49:23Z

Note: the way this works is by creating a custom asset, using the name as the public company name, the type as stock, and the note as ticker: "input"

For example:

Apple Inc
stock
ticker: AAPL

from there we can create the manual balance for the asset, which will call the apple asset we created, and then when we instantiate it with a qty, it will auto-fetch the price.

acc-no-longer-active · 2025-03-12T00:38:13Z

Have a bunch of work stuff piling on but will be free around Thursday/Friday if anyone would like to review and provide some feedback so I can take a look and do some cleanup there then!

LefterisJP · 2025-03-12T00:42:08Z

Hey @Kernlog thanks a lot for this work!

This is an over 3k PR. We will try to take a look but it may take some time and will most probably requires lots of rounds of back and forth between us to get merged.

Just mentioning to set the expectations straight. And also responding to make sure you don't think we forgot.

LefterisJP · 2025-03-12T00:43:12Z

Also in general don't merge develop on the branch but rebase the branch on develop. Though most likely we would end up squashing in the end.

acc-no-longer-active · 2025-03-12T00:58:48Z

Sounds good thanks for the note on rebasing instead @LefterisJP ! Totally understand aswell 3k PR is a good chunk to go through especially when it's not too important atm. Just wanted to mention the timings I have available this week incase someone wanted that back and fourth so they knew if I would be able to work on it or not 😆 . Ty for the response appreciate it

LefterisJP

So I gave it a quick look and I think it needs a lot of changes and should also be split in multiple PRs. The major things I noticed are:

Asset type

You seem to dynamically try to figure out the yahoo_finance thing each time from the asset by looking at the asset's notes. I think this is a hacky way to make it work with our current codebase, but the point is to do it properly.

A proper way would be to add a new asset type for stocks and ETFs. It would be very much like the other assets but would just need to be an asset or an ETF whose price can be determined automatically.

That asset would need to have as symbol the unique ticker of the ETF or stock. Is it the ISIN symbol that is checked by yahoofinance? Then at addition time the api would check it exists and has a price in yahoofinance and if yes would accept and create the asset.

This alone can be one PR.

Manual balances and price queries.

At that point you won't need any changes in the manual balances. What you did in this PR is extremely hacky and would not go in the codebase. To check the attributes dynamically and store it like this.

I understand from our earlier discussions that the reason you did it like this is that sometimes yahoo finance borks. And then may return no price. At that point you want to re-use the last valid price we have for the asset.

There is no need to have it in this hacky way. All this special logic you added in the inquirer needs to go away. It's way too hacky and unmaintainable.

What needs to happen is a more general adjustment for price queries.

That also should be its own separate PR

The query prices endpoitn should try to see if there is a cached price in the DB for the asset within the last X hours/days and return that. For each price we should probably also return its age in the prices response. So perhaps an int depending on how old the price (in seconds) is. 0 for current.

LefterisJP · 2025-03-12T09:51:57Z

rotkehlchen/balances/manual.py

+                            )
+                            price_value = entry.amount
+
+                        from rotkehlchen.types import Price


never do imports in the middle of the code. Always at top of the file

LefterisJP · 2025-03-12T09:53:21Z

rotkehlchen/inquirer.py

@@ -308,6 +314,11 @@ class Inquirer:
    special_tokens: set[str]
    weth: EvmToken
    usd: FiatAsset
+
+    # Persistent storage for custom asset prices to avoid them being wiped out
+    _custom_asset_prices: dict[str, Price] = {}


there is no need for all that. The cache is the DB itself. You just have to properly use it.

LefterisJP · 2025-03-12T09:53:42Z

rotkehlchen/inquirer.py

            manualcurrent: Optional['ManualCurrentOracle'] = None,
            msg_aggregator: Optional['MessagesAggregator'] = None,
    ) -> 'Inquirer':
+        """Create or get the already existing singleton.
+
+        In case of multi processing initialization avoid using `clear`


We never add code in production that is useful only for tests

LefterisJP · 2025-03-12T09:54:59Z

rotkehlchen/inquirer.py

+                            asset=from_asset.identifier,
+                            error=str(e),
+                        )
+            except Exception:


you should never do naked except catches in python code. Always know what exceptions your code raises.

LefterisJP · 2025-03-12T09:55:47Z

rotkehlchen/inquirer.py

        for from_asset in from_assets:
+            # Special handling for custom assets (stock/ETF)
+            try:
+                if hasattr(from_asset, 'get_asset_type') and from_asset.get_asset_type() == AssetType.CUSTOM_ASSET:


to recognize what type it is we have resolve_asset_to_class. No need to be doing hacky hasattr everywhere

LefterisJP · 2025-03-12T09:56:22Z

rotkehlchen/inquirer.py

@@ -692,7 +781,60 @@ def _preprocess_assets_to_query(
        and a list of assets that still need their prices queried.
        """
        found_prices, replaced_assets, unpriced_assets = {}, {}, []
+
+        # Special handling for custom assets to ensure we use cached prices


all this needs to go away. As i wrote in the review comment.

LefterisJP · 2025-03-12T10:16:30Z

rotkehlchen/inquirer.py

+                log.error(f"Background stock/ETF refresh process failed: {e}")
+
+        # Start the background thread
+        thread = threading.Thread(target=refresh_all_stocks_bg)


no no no don't do that. Do not use threading in rotki. Anyway as i said all of these changes need to go away

Kernlog added 7 commits March 8, 2025 19:24

feat(oracle): create yahoo oracle integrations

6737776

feat(oracle): create yahoo oracle & update inquirer/class/frontend

31d052c

feat(oracle): yahoo finance svg

957dc5f

feat(oracle): add yahoo finance to priceHistorian

821de1e

feat(oracle): create symbol recognition for yahoo finance stocks

580288f

feat(oracle): update inquirer and get oracle working correctly with c…

a272d0d

…ustom assets / manual balances

Merge branch 'rotki:develop' into yahoo-oracle

8d9af1e

feat(oracle): update readme to include stocks/etfs

a13a800

Merge branch 'rotki:develop' into yahoo-oracle

9faf10f

LefterisJP reviewed Mar 12, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Implement Yahoo Finance API for Stock/ETF Price Oracle #9227 #9587

Implement Yahoo Finance API for Stock/ETF Price Oracle #9227 #9587

Uh oh!

acc-no-longer-active commented Mar 10, 2025

Uh oh!

CLAassistant commented Mar 10, 2025 •

edited

Loading

Uh oh!

acc-no-longer-active commented Mar 10, 2025

Uh oh!

acc-no-longer-active commented Mar 12, 2025

Uh oh!

LefterisJP commented Mar 12, 2025

Uh oh!

LefterisJP commented Mar 12, 2025

Uh oh!

acc-no-longer-active commented Mar 12, 2025

Uh oh!

LefterisJP left a comment

Uh oh!

LefterisJP Mar 12, 2025

Uh oh!

LefterisJP Mar 12, 2025

Uh oh!

LefterisJP Mar 12, 2025

Uh oh!

LefterisJP Mar 12, 2025

Uh oh!

LefterisJP Mar 12, 2025

Uh oh!

LefterisJP Mar 12, 2025

Uh oh!

LefterisJP Mar 12, 2025

Uh oh!

Uh oh!

Uh oh!

Implement Yahoo Finance API for Stock/ETF Price Oracle #9227 #9587

Are you sure you want to change the base?

Implement Yahoo Finance API for Stock/ETF Price Oracle #9227 #9587

Uh oh!

Conversation

acc-no-longer-active commented Mar 10, 2025

Yahoo Finance API Integration

Implementation Overview

Major Files Updated

New Files

Modified Files

Integration Details

Yahoo Finance API Integration

Custom Asset Integration

Inquirer Integration

Manual Balances Integration

Testing

Future Improvements

Related Issues

Uh oh!

CLAassistant commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

acc-no-longer-active commented Mar 10, 2025

Uh oh!

acc-no-longer-active commented Mar 12, 2025

Uh oh!

LefterisJP commented Mar 12, 2025

Uh oh!

LefterisJP commented Mar 12, 2025

Uh oh!

acc-no-longer-active commented Mar 12, 2025

Uh oh!

LefterisJP left a comment

Choose a reason for hiding this comment

Asset type

Manual balances and price queries.

Uh oh!

LefterisJP Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

LefterisJP Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

LefterisJP Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

LefterisJP Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

LefterisJP Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

LefterisJP Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

LefterisJP Mar 12, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

CLAassistant commented Mar 10, 2025 •

edited

Loading