Trying to use selenium to expand rows in a table but it doesn't seem to work

Question

0.00/5 (No votes)

See more:

C#

  1  public async Task<string> GetHtmlSelenium(string url)
  2  {
  3      string html = "";
  4      ChromeOptions options = new ChromeOptions();
  5      //options.AddArgument("--headless");
  6      options.AddArgument("--disable-gpu");
  7      options.AddArgument("--no-sandbox");
  8      options.AddArgument("--ignore-certificate-errors"); 
  9      options.AddArgument("--enable-logging");
 10  
 11      using (IWebDriver driver = new ChromeDriver(options))
 12      {
 13          driver.Navigate().GoToUrl(url);
 14  
 15          IList<IWebElement> parentElements = driver.FindElements(By.CssSelector(".competition_row.clickable"));
 16  
 17          foreach (IWebElement parentElement in parentElements)
 18          {
 19              try
 20              {
 21                  parentElement.Click();
 22              }
 23              catch (Exception ex) { }
 24  
 25              var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(25));
 26              wait.Until(ExpectedConditions.ElementIsVisible(By.Id("c87")));
 27              string pageSource = driver.PageSource;
 28          }
 29      }
 30      return html;
 31  }

What I have tried:

I'm trying to scrape the schedule from www.livesoccertv.com/schedules/2023-08-20/ and I can get the top half of the page fine, but once you get halfway down (to the line "Africa - CAF Confederation Cup") then the table is collapsed and the data isn't in the html. The code I've posted seems to move through the elements fine as I can see it scrolling down the page but it never actually loads the data. What am I doing wrong? Thanks

Posted 24-Aug-23 0:34am

punk_legend

Updated 24-Aug-23 2:30am

Richard Deeming

v2

Add a Solution

Comments

Richard Deeming 24-Aug-23 8:34am

We get a lot of "watch sports-ball online!" spam, and a link to a "live soccer TV" site could make your question look like more of the same.

I've removed the link, but left the URL, since the markup of that specific page is pertinent to your question.

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Andre Oosthuizen · Answer 1 · 2023-08-24T00:57:00

If I look at the link up to the table row 'Africa - CAF..', it has a span id of 'c88' -

HTML

<div class="competition_row clickable" onclick="showMatches('88','2023-08-20','Live')"><a href="/competitions/international/caf-confederation-cup/" class="flag eurl" title="Go to CAF Confederation Cup page"> </a>Africa - CAF Confederation Cup<span id="clive88" class="clive"></span></div><span id="c88"></span>

Your code is instructed to stop at 'c87' -

C#

wait.Until(ExpectedConditions.ElementIsVisible(By.Id("c87")));

Increase this and it will work. To do this dynamically, replace your single 'wait.Until(...)' statement with a loop that finds all the 'span' elements with IDs starting with 'c'. Then, for each of those elements, it clicks on them, waits until they become stale, and then adds the page source to your 'htmlList' -

C#

public async Task<List<string>> GetHtmlSelenium(string url)
{
    List<string> htmlList = new List<string>();
    ChromeOptions options = new ChromeOptions();
    //options.AddArgument("--headless");
    options.AddArgument("--disable-gpu");
    options.AddArgument("--no-sandbox");
    options.AddArgument("--ignore-certificate-errors"); 
    options.AddArgument("--enable-logging");

    using (IWebDriver driver = new ChromeDriver(options))
    {
        driver.Navigate().GoToUrl(url);

        IList<IWebElement> parentElements = driver.FindElements(By.CssSelector(".competition_row.clickable"));

        foreach (IWebElement parentElement in parentElements)
        {
            try
            {
                parentElement.Click();
            }
            catch (Exception ex) { }

            var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(25));

            //Find all span elements with IDs starting with 'c'...
            IList<IWebElement> spanElements = driver.FindElements(By.CssSelector("span[id^='c']"));

            foreach (IWebElement spanElement in spanElements)
            {
                try
                {
                    wait.Until(ExpectedConditions.ElementToBeClickable(spanElement));
                    spanElement.Click();
                    wait.Until(ExpectedConditions.StalenessOf(spanElement));
                    htmlList.Add(driver.PageSource);
                }
                catch (Exception ex) { }
            }
        }
    }
    return htmlList;
}