the cluster refine codes in on refine_cluster branch

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

the cluster refine codes in on refine_cluster branch

Yongming Zhao
20 hours ago, Weijin pushed out our first effort of forward porting our cluster refine codes, in this big fat patch, we have refined the cluster communication in the message level, to archive better performance.

what we do:
        • make cluster a pure message driven layer, no more vc splice on each side
        • cleanup the msg encapsulation and callback implements
        • modified the cache cluster interface
due to the big change we made, there is something we changed in the cluster:
        * load monitor
        * hostdb cluster interface
as our main platform is Linux, and due to the network codes we can not reuse, we have made some dirty codes into the new cluster, that will need more work to get clean, to make it multi-platform aware and clean.

anyway, it is out, please join us on the hacking and testing.

those codes performs >4G/box traffic in our network, and there is no limit in cluster performance anymore

please refer to the codes and wiki:
codes in:
https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;a=shortlog;h=refs/heads/refine_cluster
wiki at:
https://cwiki.apache.org/confluence/display/TS/Clustering
jira at:
https://issues.apache.org/jira/browse/TS-2005

the patch is a joint effort of WeiJin and YuQing
thanks



Yongming Zhao
赵永明
aka 永豪 [hidden email]


signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: the cluster refine codes in on refine_cluster branch

Neddy, NH. Nam
Thanks Yongming, but do you have archive doc/guide to set up cluster?


On Thu, Nov 21, 2013 at 1:45 PM, Yongming Zhao <[hidden email]> wrote:
20 hours ago, Weijin pushed out our first effort of forward porting our cluster refine codes, in this big fat patch, we have refined the cluster communication in the message level, to archive better performance.

what we do:
        • make cluster a pure message driven layer, no more vc splice on each side
        • cleanup the msg encapsulation and callback implements
        • modified the cache cluster interface
due to the big change we made, there is something we changed in the cluster:
        * load monitor
        * hostdb cluster interface
as our main platform is Linux, and due to the network codes we can not reuse, we have made some dirty codes into the new cluster, that will need more work to get clean, to make it multi-platform aware and clean.

anyway, it is out, please join us on the hacking and testing.

those codes performs >4G/box traffic in our network, and there is no limit in cluster performance anymore

please refer to the codes and wiki:
codes in:
https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;a=shortlog;h=refs/heads/refine_cluster
wiki at:
https://cwiki.apache.org/confluence/display/TS/Clustering
jira at:
https://issues.apache.org/jira/browse/TS-2005

the patch is a joint effort of WeiJin and YuQing
thanks



Yongming Zhao
赵永明
aka 永豪 [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: the cluster refine codes in on refine_cluster branch

Igor Galić-2

Thanks Yongming, but do you have archive doc/guide to set up cluster?



--
Igor Galić

Tel: +43 (0) 664 886 22 883
Mail: [hidden email]
URL: http://brainsware.org/
GPG: 8716 7A9F 989B ABD5 100F  4008 F266 55D6 2998 1641

Reply | Threaded
Open this post in threaded view
|

Re: the cluster refine codes in on refine_cluster branch

Yongming Zhao
yeah, that is the origin docs for cluster, I will update all the config changes we done into the wiki tomorrow, for the refined cluster.

thanks

在 2013年11月21日,下午6:01,Igor Galić <[hidden email]> 写道:


Thanks Yongming, but do you have archive doc/guide to set up cluster?



--
Igor Galić

Tel: +43 (0) 664 886 22 883
Mail: [hidden email]
URL: http://brainsware.org/
GPG: 8716 7A9F 989B ABD5 100F  4008 F266 55D6 2998 1641






signature.asc (817 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: the cluster refine codes in on refine_cluster branch

James Peach
In reply to this post by Yongming Zhao
On Nov 20, 2013, at 10:45 PM, Yongming Zhao <[hidden email]> wrote:

> 20 hours ago, Weijin pushed out our first effort of forward porting our cluster refine codes, in this big fat patch, we have refined the cluster communication in the message level, to archive better performance.
>
> what we do:
> • make cluster a pure message driven layer, no more vc splice on each side
> • cleanup the msg encapsulation and callback implements
> • modified the cache cluster interface
> due to the big change we made, there is something we changed in the cluster:
> * load monitor
> * hostdb cluster interface
> as our main platform is Linux, and due to the network codes we can not reuse, we have made some dirty codes into the new cluster, that will need more work to get clean, to make it multi-platform aware and clean.
>
> anyway, it is out, please join us on the hacking and testing.
>
> those codes performs >4G/box traffic in our network, and there is no limit in cluster performance anymore
>
> please refer to the codes and wiki:
> codes in:
> https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;a=shortlog;h=refs/heads/refine_cluster
> wiki at:
> https://cwiki.apache.org/confluence/display/TS/Clustering
> jira at:
> https://issues.apache.org/jira/browse/TS-2005
>
> the patch is a joint effort of WeiJin and YuQing

Thanks guys, and thanks for writing up the changes. I'll try to review over the next few weeks and ask a lot of questions.

One think I'd appreciate a lot is some guidance on how to test clustering. Do you have advice on setting up a performance benchmark for testing the cluster? Do you have any measurements from your own testing that you can share?

J
Reply | Threaded
Open this post in threaded view
|

Re: the cluster refine codes in on refine_cluster branch

Yongming Zhao
In reply to this post by Yongming Zhao
https://cwiki.apache.org/confluence/display/TS/Clustering

here is the long waiting documents on how to config and manage the new cluster, and I will translate them all into English.

the merge codes may have some issue, on building and performance, please don’t put in production anyway.

thanks


在 2013年11月21日,下午2:45,Yongming Zhao <[hidden email]> 写道:

> 20 hours ago, Weijin pushed out our first effort of forward porting our cluster refine codes, in this big fat patch, we have refined the cluster communication in the message level, to archive better performance.
>
> what we do:
> • make cluster a pure message driven layer, no more vc splice on each side
> • cleanup the msg encapsulation and callback implements
> • modified the cache cluster interface
> due to the big change we made, there is something we changed in the cluster:
> * load monitor
> * hostdb cluster interface
> as our main platform is Linux, and due to the network codes we can not reuse, we have made some dirty codes into the new cluster, that will need more work to get clean, to make it multi-platform aware and clean.
>
> anyway, it is out, please join us on the hacking and testing.
>
> those codes performs >4G/box traffic in our network, and there is no limit in cluster performance anymore
>
> please refer to the codes and wiki:
> codes in:
> https://git-wip-us.apache.org/repos/asf?p=trafficserver.git;a=shortlog;h=refs/heads/refine_cluster
> wiki at:
> https://cwiki.apache.org/confluence/display/TS/Clustering
> jira at:
> https://issues.apache.org/jira/browse/TS-2005
>
> the patch is a joint effort of WeiJin and YuQing
> thanks
>
>
>
> Yongming Zhao
> 赵永明
> aka 永豪 [hidden email]
>


signature.asc (817 bytes) Download Attachment