|
I have met the following problem. I have a pipeline that reads data from an MS SQL Server and stores them into a file in a BLOB container in Azure Storage. The file has Parquet (or Apache Parquet, as it is also called) format.
So, when the “sink” (output) file is stored in a compressed way (snappy, or gzip – does not matter) AND the file is large enough (more than 50 Mb), the pipeline failed. The message was the following:
"errorCode": "2200",
"message": "Failure happened on 'Sink' side. ErrorCode=UserErrorJavaInvocationException,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=An error occurred when invoking java, message: java.lang.OutOfMemoryError:Java heap space\ntotal entry:11\r\njava.util.ArrayDeque.doubleCapacity(Unknown Source)\r\njava.util.ArrayDeque.addFirst(Unknown Source)\r\njava.util.ArrayDeque.push(Unknown Source)\r\norg.apache.parquet.io.ValidatingRecordConsumer.endField(ValidatingRecordConsumer.java:108)\r\norg.apache.parquet.example.data.GroupWriter.writeGroup(GroupWriter.java:58)\r\norg.apache.parquet.example.data.GroupWriter.write(GroupWriter.java:37)\r\norg.apache.parquet.hadoop.example.GroupWriteSupport.write(GroupWriteSupport.java:87)\r\norg.apache.parquet.hadoop.example.GroupWriteSupport.write(GroupWriteSupport.java:37)\r\norg.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:123)\r\norg.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:292)\r\ncom.microsoft.datatransfer.bridge.parquet.ParquetBatchWriter.addRows(ParquetBatchWriter.java:60)\r\n,Source=Microsoft.DataTransfer.Common,''Type=Microsoft.DataTransfer.Richfile.JniExt.JavaBridgeException,Message=,Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'",
"failureType": "UserError",
"target": "Work_Work"
}
The "Work_Work" is name of a Copy Data activity in the pipeline.
If I turn the compression off (the generated BLOB file is uncompressed), the error does not happen.
Is this the error described in https://docs.microsoft.com/en-us/azure/data-factory/format-parquet: the “…If you copy data to/from Parquet format using Self-hosted Integration Runtime and hit error saying "An error occurred when invoking java, message: java.lang.OutOfMemoryError:Java heap space", you can add an environment variable _JAVA_OPTIONS in the machine that hosts the Self-hosted IR to adjust the min/max heap size for JVM to empower such copy, then rerun the pipeline….”?
If it is, have I understood correctly that I have to do the following:
To go to a server where the “Self-hosted Integration Runtime” (still have no idea what it is) and increase the max heap size for JVM. Is this correct?
If it is, my next question is: how large the max heap size should be? My pipeline can generate a file whose size will be 30 GB.
What “max heap size” can guarantee that such a file will not cause the fail?
|
|
|
|
|
Hello everyone,
I am having a question. I want to make a website scanner that allows me to scan the whole website for one word or one color. This because there isn't one available (at least i couldn't find one).
I am not a programming wonder and i really hope or you could help me. I would really appreciate it.
Thanks in advance!
Regards,
Lars
|
|
|
|
|
You need a web scraping solution, something like this in .NET world, Html Agility pack | Html Agility Pack.
You can find out other packages in other languages and runtimes too if you find. The basic concept is you download the entire web page as HTML document and this tool parses it down. You then read the document and look for your "word or color". That is the closest of what I can think of.
The sh*t I complain about
It's like there ain't a cloud in the sky and it's raining out - Eminem
~! Firewall !~
|
|
|
|
|
To find a specific word or phrase on a web page on your computer
open a webpage in Chrome.
At the top right, click More More and then Find.
Type your search term in the bar that appears in the top right.
Press Enter to search the page.
Matches appear highlighted in yellow. You can see where all the matches are located on a webpage using the yellow markers on the scrollbar.
I hope this helps!
Ben Martin
|
|
|
|
|
Visit www.google_com/ in your browser.
Enter site: followed by the site address you want to search.
Enter the word or phrase you want to search for after the site address.
Regards,
Peter
|
|
|
|
|
part rant, part some details I have found so far.
I am trying to figure out what the basic cost is to run various Azure app setups.
Like a simple todo web application - connected to a very simple data store.
But when most searches just end with using the Pricing calculator, for me it is an overload of information.
Very simple - the minimum use of Cosmos DB is $23.61 per month. But then you need to add what other components??????
So if anyone knows of some resources providing to developers the minimum costing to run some small applications, that would be great?
now some info:
(most numbers based on West Europe/UK zone selection, which is most cases is the same as US pricing)
Prices are Per Month
Asure Functions: $0.00 - it takes almost 1 million 2 second calls before 1$ would be hit.
Cosmos DB: minimum 4 x100 RU/sec + 1 GB = $23.61
Storage Account (Block Blob Storage): 1GB with 1000 on all operations: £1.25
Storage Account (Block Blob Storage): 1GB with 10,000 operations: $0.11 (is this right?)
Storage Account (Table Stoage): 1GB with 100,000 transactions: $0.09
Storage Account (Data Lake gen2): 1GB with 10,000 on each operation type: $0.74
Asure SQL Database: this is over $1,699.22 per month at a minimum???
App Service: $68.62
So apart from Cosmos DB, for entry level mess about with azure for storage, AFTER FREE account expired, some seem sensible.
|
|
|
|
|
Hi all,
I have an app service that works fine for most folks. This one client is having an issue and I'm 95% sure they're sending something wrong. So I need to see exactly what payload they're sending. What's the best way to do this in Azure? I thought about Fiddler but doesn't that need to sit on the same machine as the webserver? Anyway... if you could give me some ideas on how I can see this payload I'd greatly appreciate it.
Denise
Hypermommy
|
|
|
|
|
Hypermommy wrote: I thought about Fiddler but doesn't that need to sit on the same machine as the webserver?
No, Fiddler sits on the client computer.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
That is exactly why you use logging. Elmah can be a good place to start if your application is .NET based. ELMAH—Home
You need to check what is being input, and maintain that as a token. If I had to do this, I would most likely maintain a trace of the exception thread, store the states, or the request object that they had sent including query strings, headers etc, or the account information, and provide the user with a token, Guid.NewGuid(), in the response along with a message stating something went wrong and ask them to reach me out with that token in order to have that fixed for their account.
Using Postman won't help at all, because you are still working on the client-side and not check what is going wrong, and exposing error information on the public internet—even for personal understanding—never results in a good way. Someone might just understand the workflow and use that for nefarious purposes.
The sh*t I complain about
It's like there ain't a cloud in the sky and it's raining out - Eminem
~! Firewall !~
|
|
|
|
|
Thanks, y'all! Now it seems I just have to learn how to get my setup right on here so I can see answers.  But seriously, I do appreciate the help.
Hypermommy
|
|
|
|
|
First, if this isn't the right forum, my apologies, and please direct me to the correct one...
I have read, and searched for a week now, and I just don't seem to get it.
I have a Microsoft Partner Center Global Admin User; "SvcUser"
Using the Partner Center API with this user, I can list all our customers, and get their tennantIds.
Using a TennantId, I gan get the customers users and subscriptions, and all is well.
The App registered in Azure have all the permissions set up...
Now I need to get all domains of a specific tennant... The Partner Center API does not support this, so I turn to Microsoft Graph.
The concept, as I understand it, is to use the "common" endpoint to log in with my "SvcUser" and get a Autherntication code. Then connect to the specific tennant endpoint, supplying the code from the previous step, to get an Authorization token.
Once I have this "Authorization token", I should be home free...
A.) Can anyone provide me with a clear example on how to implement the Authentication and Authorization calls (C# preferably  ) ?
B.) secondary; straighten me out, if all of the above is completely rubbish ?
Thanks!
|
|
|
|
|
We do not have a specific Microsoft Graph forum, neither a Partner Center one, so this one sounds good enough.
The thing is, as far as I remember, we needed to do a project similar to this one, and we realized that most of the auth was done using Azure AD. That step you are talking about is the steps for Azure AD.
For the Partner Center, check the following links for reference to see how you authenticate,
Partner Center authentication | Microsoft Docs. Once done with the authentication, you can use the same token and authenticate yourself against that token, see the headers for reference Partner Center REST headers | Microsoft Docs. The domains for the account can be checked, List domains - Documentation - Microsoft Graph and as I briefed you a bit earlier, the tokens are mostly captured using the Azure AD and so did Microsoft do that read the complete documentation here, Get auth tokens - Documentation - Microsoft Graph.
Most of the scenarios that revolve around the Partner Center are mentioned here, check if you can find your scenario here as well: Scenarios | Microsoft Docs
The sh*t I complain about
It's like there ain't a cloud in the sky and it's raining out - Eminem
~! Firewall !~
|
|
|
|
|
<pre>Hi,
I have been using Discount ASP.Net to host my site for the last 6 years+.
Really pleased with them, great support and works well and they have a UK data centre option.
I was looking at their Everleap service (built on MS Azure), which looks great as it isolates you from most of the brain ache of having a VPS or MS Azure.
But they don't have a UK data centre option for this service, big disappointment!
I am worried that if I use the west coast (I think) US based Everleap service my websites page load times will increase significantly.
So I have two questions....
1 Does anyone know of a similar service to Everleap that has a UK data centre option?
2 Are my concerns regarding increased page load times of a west coast US hosted site overblown?
I already use a CDN for all my static content.
Many thanks to everyone in advance,
Will</pre>
|
|
|
|
|
Will Stanley wrote: I am worried that if I use the west coast (I think) US based Everleap service my websites page load times will increase significantly.
Yes, they will increase, but not significantly. There is a minor increase in the page loading, and after all that will only affect if you are building a real time application or have to respond back in less then 10ms. Most of Azure services guarantee less than that response.
I would personally state that the fact is a bit overoverblown, as the fact that you are also using CDN services, then your service will be cached around the CDN network itself. If you are not expecting a real time solution, then I suggest this service is enough and you can lift the performance or decrease the response time using the CDN providers (that you already have).
But I am not an EverLeap user and I am personally inclined towards Azure, so please wait until someone else also answers the question to compare the reviews.
The sh*t I complain about
It's like there ain't a cloud in the sky and it's raining out - Eminem
~! Firewall !~
|
|
|
|
|
Hi,
Many thanks for the reply.
I think I just have to do a 30 day trial.
Will
|
|
|
|
|
When a Azure-WebJob fails at startup/runtime i can get thos Information also saved in the Azure-file system at data/jobs/continuous/jobName.
How can i get this Information/File in C# to read the information (success/failed) ?
Best Regards
Nicole 
|
|
|
|
|
|
1: never spend all day Sunday doing this.
2: Help!
3: Serves me right: got lost in the weeds...
Ok, so I have setup my azure web site. Pretty straightforward, got the SSL working and all is sweet, works like a charm. Now for the bit I am struggling with...
Let's say my main site is called www.robots.life (a legit domain, btw).
Now I want to do admin.robots.life. I have setup the CNAME record and azure is fine with admin.robots.life as an application setup as /Admin, site/admin (where the files reside).
I have also added a rewrite rule to send users to admin/robots.life.
But it is not working. Firstly, where do I physically locate the files? I have it as site/Admin and robots.life, the main site, as site/wwwroot. Is that correct?
It appears to be displaying a path that has an extra "Admin" and goes straight to a blank page. Nothing appears in the azure log or my log to show what the error might be and switching custom errors off does nothing. It does get to the login page if I enter the url manually but then hangs after entering user id, password and submitting.
I can only imagine I am doing something monumentally stupid. Like breathing.
Would really appreciate some direction here. If this isn't enough info, be happy to supply several reams of closely scrawled gibberish.
Thanks
|
|
|
|
|
|
That was the easy bit. Getting it to work appears somewhat more difficult.
Thanks for the response though.
Anyway, after a night away from it, I'll look again when I get home. I have a feeling it's me being as dumb as a box of rusty spanners that's the problem.
|
|
|
|
|
|
Is that supposed to mean something?
|
|
|
|
|
Done. Thank you.
There are only 10 types of people in the world, those who understand binary and those who don't.
|
|
|
|
|
i have lots of doubt in deduplication in cloud storage anyone please guide and help me 
|
|
|
|
|
If you want help, then you need to ask a question.
"These people looked deep within my soul and assigned me a number based on the order in which I joined."
- Homer
|
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
