dataframe or sqlctx (sqlcontext) generated "Trying to call a package" error

I am using spark 1.3.1. In PySpark, I have created a DataFrame from a RDD and registered the schema, something like this :

dataLen=sqlCtx.createDataFrame(myrdd, ["id", "size"])
dataLen.registerTempTable("tbl")

at this point everything is fine I can make a "select" query from "tbl", for example "select size from tbl where id='abc'".

Then in a Python function, I define something like :

def  getsize(id):
    total=sqlCtx.sql("select size from tbl where id='" + id + "'")
    return total.take(1)[0].size

at this point still no problem, I can do getsize("ab") and it does return a value.

The problem occurred when I invoked getsize within a rdd, say I have a rdd named data which is of (key, value) list, when I do

data.map(lambda x: (x[0], getsize("ab"))

this generated an error which is

py4j.protocol.Py4JError: Trying to call a package

Any idea?

Solved

Spark doesn't support nested actions or transformations and SQLContext is not accessible outside the driver. So what you're doing here simply cannot work. It is not exactly clear what you want here but most likely a simple join, either on RDDs or DataFrames should do the trick.

Showing multiple PDF Files in WebBrowser control (Windows Form)

I’m trying to show PDF files in Windows Form WebBrowser control. I do this by WebBrowser.Navigate() method. It is work when I want show one PDF file, which is not enough for me. I need when a user clicks a button , WebBrowser show another PDF file. What I have to do?

Solved

Avoid using WebBrowser control just to show a pdf. Try using one of the many free PDF libraries available for .net such as PDFSharp.

This will allow you to draw pdf's on to the Form and using TabControl you can create tabs programmatically and place a new pdf viewer inside of the tab.

You could easily make a 2 different files PDF viewer (side by side or top and bottom) by using the PDFSharp. Or you can use the ActiveX control for Adobe Reader.

Edit: It was my understanding that PDFSharp can render pdf's directly. This can only be done by exporting the page to a Jpeg/Png and then displaying the images. Sorry for any confusion.

At your place I'd do tabControl, and foreach tabControl, I'd put a webbroswer inwhich there would be a PDF by Tab.

EDIT : A WebBroswer isn't like internet explorer. Internet Explorer has the "Feature" of having multiple tabs (just like Chrome, etc). The WebBroswer Element is only the content of the tab :)

EDIT2 : After many years into this, I would recommend using a PDF library also. As a prototype usecase it might work, but in real world, we never know how many there will be. the best would probably to: either let the option to the user to open in a competent software i.e. Acrobat Reader where tab is natively used (DC), or use a PDF Library.

What i think is that you are facing a very simple navigate issue here. I have worked out a project for you, take a look at it, I hope it will solve your issue. download here.

AxisFault at org.apache.axis2.AxisFault.makeFault

I have an error when running axis client, i'm using axis2 version 1.7.7

org.apache.axis2.AxisFault
at org.apache.axis2.AxisFault.makeFault(AxisFault.java:430)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:123)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:67)
at org.apache.axis2.description.OutInAxisOperationClient.handleResponse(OutInAxisOperation.java:354)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:417)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:229)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:165)
at ch.e_dec.www.services.edecservice.v2.ServiceStub.goodsDeclarations(EdecServiceStub.java:339)
at ch.e_dec.www.services.edecservice.EdecClient.main(EdecClient.java:318)
Caused by: java.lang.NullPointerException
at org.apache.axis2.builder.MIMEBuilder.processDocument(MIMEBuilder.java:78)
at org.apache.axis2.transport.TransportUtils.createDocumentElement(TransportUtils.java:197)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:145)
at org.apache.axis2.transport.TransportUtils.createSOAPMessage(TransportUtils.java:108)

I have checked with axis2 1.4.1 version , there is no issue found , And for upgrading axis version same code has done with latest version 1.7.7

In 1.4.1 version it was successful,

contentType:Before:multipart/related; boundary="----=_Part_3837_17859151.1531477242209"
contentType:After:multipart/related; boundary="----=_Part_3837_17859151.1531477242209"; type="text/xml"

The type is getting as type="text/xml" while using axis2 1.4.1

but in axis2 1.7.7 , when i checked it is found that the content type is null . so null pointer exception is coming . i want to know why the type is getting as null

contentType:Before:multipart/related; boundary="----=_Part_3837_17859151.1531477242209"
contentType:After:multipart/related; boundary="----=_Part_3837_17859151.1531477242209"; 

please help me.

Solved

Backward incompatible changes as described in Axis2 1.7.0 release notes:

Apache Axis2 1.7.0 is a major release that contains lots of bug fixes and improvements, some of which introduce backward incompatible changes with respect to Axis2 1.6.x (as described below).

More specifically:

.. application code should no longer assume that the Attachments object is set on the message context and contains all MIME parts of the message. Note that as of Axis2 1.7.0, this assumption is still true (because the JAX-WS implementation internally relies on this), but this will change in a subsequent release.

Class apache/axis2/builder/MIMEBuilder.java is mentioned on this JIRA issue.

You have to refactor your implementation to be able to upgrade the version.

Where is C++ /Zo in Visual Studio 2013 Update 3

http://www.visualstudio.com/news/2014-aug-4-vs

Mentions a new compiler switch for better debugging with optimizations on but I can't for the life of me find it.

Does anyone know where it is and how to turn it on?

C++

/Zo Compiler Switch

/Zo is a compiler switch that generates richer debugging information for optimized code (non /Od builds). This includes improved support for debugging local variables. (Minimum edition: Express)

Solved

See the documentation on MSDN here: /Zo (Enhance Optimized Debugging). For command-line builds, just add /Zo to the build commands. To add /Zo to a project in Visual Studio, open the Property Pages dialog for the project, and navigate to Configuration Properties > C/C++ > Command Line in the tree view control. Add /Zo to the Additional Options edit box, then click OK. You should be good to go. New options like this usually don't get added to the Project Properties dialog UI until the next major revision, so you have to use the Additional Options box to see them in action.

Note that there are some incompatible options such as native Edit and Continue, but that's an unusual thing to do in Release mode anyway.