Ah, lazy loading. As someone working on a dinky, 4G RAM laptop whose hobby is "throttling myself to 33% CPU if i start overheating", I understand. It must be done. Shopping websites need to find the middle ground between showing you their entire stock and crashing your browser.
When I started using requests and BeautifulSoup4, everything felt so simple. (import antigravity) Reading guides and watchig walkthroughs to better understand, I found it funny how, in many cases, they would pause the python lesson to introduce the concept of HTML tags and attributes. Considering I started my programming journey with early HTML and CSS, it's sort of funny to realise this isn't the default for most people. But, the modern HTML/CSS environment is not the one I learned in 2002. This is obvious, but it still is somehow a bit of surprise to me. I am constantly surprised to find that so-and-so was deprecated a decade ago, even though I haven't used it myself in just as long.
Anyway. Selenium. Thanks, python community, for making things so, so easy to understand. There are 1000 things that are simple to do because someone out there made a module for it.
I've played around with these web scraping tools quite a bit. I think, given that I am careful, and I make sure to keep every site's robots.txt close to my heart, I can scrape whatever I need from any standard webpage on the internet. I will dive a bit deeper into Selenium on python. Apparently, you can also use her on javascript? Interesting. If I want to turn my project from a single python script that runs and posts results in the terminal, I will have to bite the bullet and go crawling back to Node.JS
My beloathed.
The alternative is to make it a browser-based application. This is possible. I have learned how these things interact with each other, and Selenium has the functionality to grab webpage info in-browser. We'll see. Javascript is Javascript, no matter where it lands.
Wish me luck.