Click here to Skip to main content
15,886,199 members
Please Sign up or sign in to vote.
1.00/5 (1 vote)
hi i want to scrap all the pairs that kucoin support from the webpage but the code that i wrote just return a empty list how should i correct it.
thank you so much.

What I have tried:

Python
from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome(executable_path = r'C:\Users\Computer\Desktop\data\chromedriver.exe')
driver.get('https://www.kucoin.com/markets?lang=en_US')
results = []
content = driver.page_source
soup = BeautifulSoup(content)
driver.quit()
for element in soup.findAll(attrs = "tdHeadBox___2P8mW"):
    name = element.find('span')
    if name not in results:
        results.append(name.text)

print(results)
Posted
Updated 11-Sep-21 13:53pm
v2
Comments
Dave Kreskowiak 11-Sep-21 20:10pm    
You could be running into a situation where the data you want is not part of the page you download, but is instead populated by javascript code. You'll have to check the content of the page you downloaded to find out.

1 solution

Quote:
the code that i wrote just return a empty list how should i correct it.

You have 1 large problem, you have no idea what is wrong, your code is a black box to you.
First thing, split the problem to see where içs go wrong.
Python
from bs4 import BeautifulSoup
from selenium import webdriver

driver = webdriver.Chrome(executable_path = r'C:\Users\Computer\Desktop\data\chromedriver.exe')
driver.get('https://www.kucoin.com/markets?lang=en_US')
results = []
content = driver.page_source
soup = BeautifulSoup(content)
driver.quit()
# at this point, add code to save the page in text file, so that you can check that you have the expected page. 
# then, if page is correct, check if the string you search is in the page
for element in soup.findAll(attrs = "tdHeadBox___2P8mW"):
    # print/save every element found    
    name = element.find('span')
    if name not in results:
        results.append(name.text)

print(results)

Your code do not behave the way you expect, or you don't understand why !

There is an almost universal solution: Run your code on debugger step by step, inspect variables.
The debugger is here to show you what your code is doing and your task is to compare with what it should do.
There is no magic in the debugger, it don't know what your code is supposed to do, it don't find bugs, it just help you to by showing you what is going on. When the code don't do what is expected, you are close to a bug.
To see what your code is doing: Just set a breakpoint and see your code performing, the debugger allow you to execute lines 1 by 1 and to inspect variables as it execute.

Debugger - Wikipedia, the free encyclopedia[^]

Mastering Debugging in Visual Studio 2010 - A Beginner's Guide[^]
Basic Debugging with Visual Studio 2010 - YouTube[^]

<
27.3. pdb — The Python Debugger — Python 3.6.1 documentation[^]
Debugging in Python | Python Conquers The Universe[^]
pdb – Interactive Debugger - Python Module of the Week[^]

The debugger is here to only show you what your code is doing and your task is to compare with what it should do.
 
Share this answer
 

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900