Skip to main content

web access - Using URLExecute to login to website



One of my students is doing summer research for a local company to help analyze some of their data. Trouble is, the web interface they provide for downloading the data only allows one to download one customer's info at a time; there are over 3000 customers to analyze.


I'd like to use Mathematica (MMA) to automate this process. Step 1: use MMA to log in to their website. I've looked at many similar posts and still struggle to understand how to use Import or URLFetch or URLExecute to accomplish this.


I can't share the company's URL, though from the page source this seems to be the relevant code:






I tried using


URLExecute["https://THEIR_WEB_ADDRESS.com/src/index.php?page=admin", {"username" -> "fooUser", "password" -> "fooPassword"}]


but the response seems to indicate the website doesn't understand the requested page. I also tried variations of "Username" vs "username", etc., to no avail.


Can someone point me in the right direction based on the page source? I know you won't be able to test your own answer as the actual URL is not given.



Answer



This is getting a bit too long for a comment. What you want to do is possible, in principle, but web servers can be picky about how the request should look. What we can do is to try to provide as much information as possible to help Mathematica make an acceptable request. I would start with this:


req = HTTPRequest[

   "https://THEIR_WEB_ADDRESS.com/src/index.php", <|
    Method -> "POST",
    "Query" -> {"page" -> "login"},
    "Body" -> {
      "username" -> "fooUser",
      "password" -> "fooPassword",
      "action" -> "login"
      },
    "ContentType" -> "application/x-www-form-urlencoded"
    |>];


resp = URLRead[req]

The action parameter comes from the HTML for the form.


It would be interesting to know what resp["StatusCode"] returns. If it returns OK, it would also be good to check resp["Cookies"] to see if it returned a cookie. And, of course, you can also check resp["Body"] to see what they sent back.


Please let me know how it goes.


Comments

Post a Comment

Popular posts from this blog

functions - Get leading series expansion term?

Given a function f[x] , I would like to have a function leadingSeries that returns just the leading term in the series around x=0 . For example: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x)] x and leadingSeries[(1/x + 2 + (1 - 1/x^3)/4)/(4 + x)] -(1/(16 x^3)) Is there such a function in Mathematica? Or maybe one can implement it efficiently? EDIT I finally went with the following implementation, based on Carl Woll 's answer: lds[ex_,x_]:=( (ex/.x->(x+O[x]^2))/.SeriesData[U_,Z_,L_List,Mi_,Ma_,De_]:>SeriesData[U,Z,{L[[1]]},Mi,Mi+1,De]//Quiet//Normal) The advantage is, that this one also properly works with functions whose leading term is a constant: lds[Exp[x],x] 1 Answer Update 1 Updated to eliminate SeriesData and to not return additional terms Perhaps you could use: leadingSeries[expr_, x_] := Normal[expr /. x->(x+O[x]^2) /. a_List :> Take[a, 1]] Then for your examples: leadingSeries[(1/x + 2)/(4 + 1/x^2 + x), x] leadingSeries[Exp[x], x] leadingSeries[(1/x + 2 + (1 - 1/x...
Post a Comment
Read more

How to thread a list

I have data in format data = {{a1, a2}, {b1, b2}, {c1, c2}, {d1, d2}} Tableform: I want to thread it to : tdata = {{{a1, b1}, {a2, b2}}, {{a1, c1}, {a2, c2}}, {{a1, d1}, {a2, d2}}} Tableform: And I would like to do better then pseudofunction[n_] := Transpose[{data2[[1]], data2[[n]]}]; SetAttributes[pseudofunction, Listable]; Range[2, 4] // pseudofunction Here is my benchmark data, where data3 is normal sample of real data. data3 = Drop[ExcelWorkBook[[Column1 ;; Column4]], None, 1]; data2 = {a #, b #, c #, d #} & /@ Range[1, 10^5]; data = RandomReal[{0, 1}, {10^6, 4}]; Here is my benchmark code kptnw[list_] := Transpose[{Table[First@#, {Length@# - 1}], Rest@#}, {3, 1, 2}] &@list kptnw2[list_] := Transpose[{ConstantArray[First@#, Length@# - 1], Rest@#}, {3, 1, 2}] &@list OleksandrR[list_] := Flatten[Outer[List, List@First[list], Rest[list], 1], {{2}, {1, 4}}] paradox2[list_] := Partition[Riffle[list[[1]], #], 2] & /@ Drop[list, 1] RM[list_] := FoldList[Transpose[{First@li...
Post a Comment
Read more

front end - keyboard shortcut to invoke Insert new matrix

I frequently need to type in some matrices, and the menu command Insert > Table/Matrix > New... allows matrices with lines drawn between columns and rows, which is very helpful. I would like to make a keyboard shortcut for it, but cannot find the relevant frontend token command (4209405) for it. Since the FullForm[] and InputForm[] of matrices with lines drawn between rows and columns is the same as those without lines, it's hard to do this via 3rd party system-wide text expanders (e.g. autohotkey or atext on mac). How does one assign a keyboard shortcut for the menu item Insert > Table/Matrix > New... , preferably using only mathematica? Thanks! Answer In the MenuSetup.tr (for linux located in the $InstallationDirectory/SystemFiles/FrontEnd/TextResources/X/ directory), I changed the line MenuItem["&New...", "CreateGridBoxDialog"] to read MenuItem["&New...", "CreateGridBoxDialog", MenuKey["m", Modifiers-...
Post a Comment
Read more