python "requests" package very slow retrieving html pages

Question

0.00/5 (No votes)

See more:

I'm hoping someone else has seen this issue and knows what the problem might be. I have written some python 3.4 code to read the config.xml file of a job in from Jenkins using the requests 2.7.0 package and the URL format:

http://jenkins/job/<job name>/config.xml

All is good but it takes a good 9-12 seconds for the page to be retrieved. If I put that URL in a browser the result is instantaneous so I don't think the server or network is the problem here. I have also tried to use the pyJenkins package but again any request takes the same amount of time.

I've obviously done a Google search on this but nothing seems relevant which is suggesting I'm doing something wrong as opposed to a bug in the package itself.

I used the following code to get the page:

Python

import requests

# Get a list of all the jobs on the system
a = requests.get('http://jenkins-server/api/json', params={'pretty':'true'},
                 auth=("username", "password"))

I have also tried to access web pages on the net that do not require authentication, http and https sites and other internal web servers that we have but they are all slow.

Anyone have any ideas why this is happening or suggest how to fix it?

Posted 10-Jul-15 1:35am

DelboyJay001

Updated 22-Jun-17 22:02pm

Add a Solution

Comments

Xiao Ling 24-Jul-15 4:34am

Have you tried urllib2?

DelboyJay001 23-Jun-17 3:45am

It could be something to do with the DNS lookup that I have found before. Are you running the python code on Windows or Linux? Can you try using the IP address and see if that speeds things up?

1 solution

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

DelboyJay001 · Answer 1 · 2017-06-22T22:01:00

Try this and see if it works. You could have a DNS issue so try an IP address instead of a DNS name and check if it is faster. Failing that, there is a known problem with proxy detection in the requests library even if you have your system setup to bypass which I imagine you have in this case.

import requests

session = requests.Session()
session.trust_env = False

response = session.get('http://jenkins-server/api/json', params={'pretty':'true'},
                 auth=("username", "password")))

This code snippet comes from here and talks about the same problem:

python - requests: how to disable / bypass proxy - Stack Overflow[^]